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ABSTRACT 


The  purpose  of  this  thesis  is  to  develop  a  prototype  safety  information 
management  tool  to  capture  human  error  in  Naval  Aviation  maintenance  mishaps.  The 
Human  Factors  Analysis  and  Classification  System-Maintenance  Extension  taxonomy,  an 
effective  framework  for  classifying  and  analyzing  the  presence  of  maintenance  errors  that 
lead  to  mishaps,  incidents,  and  personal  injuries,  is  the  foundation  of  this  management 
tool.  The  target  audience  for  this  information  management  system  tool  includes  safety 
personnel,  mishap  investigators,  Aircraft  Mishap  Board  (AMB)  members,  and  analysts. 

A  review  of  three  areas  is  needed  to  produce  the  prototype:  (1)  the  collection,  use,  and 
management  of  accident  information,  (2)  human  error  theories  as  related  to  aviation 
mishaps,  and  (3)  the  design  of  an  effective  mishap  database  tool.  A  usability  study  was 
conducted  using  potential  end-users  (Naval  Aviation  Safety  Officers).  The  participants 
are  given  both  written  procedures  to  navigate  through  the  prototype  and  an  exit  survey. 

The  results  of  the  survey,  including  objective  and  subjective  responses  about  the 
prototype  are  gathered.  The  resulting  data  indicates  an  improved  version  of  the  prototype 
could  directly  lead  to  a  decreased  mishap  rate  and  overall  increased  mission  readiness 
due  to  the  training  and  analysis  opportunity  it  provides. 
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I.  INTRODUCTION 


A.  OVERVIEW 

(1)  Background 

Naval  Aviation  has  had  great  success  in  substantially  reducing  (by  half)  its  Class 
A  Flight  Mishap  (FM)  rate  in  each  successive  decade  between  1950  and  1999  (see 
Figure  1).  Despite  this  achievement,  the  proportion  of  mishaps  attributed  to  human  error 
has  remained  at  a  relative  constant  rate  of  about  four  in  five  (Nutwell  &  Sherman,  1997). 
In  1996,  a  Navy  F-14  Tomcat  crashed  shortly  after  taking  off  from  Nashville,  Tennessee 
killing  both  aircrew  and  three  civilians  on  the  ground  (HFQMB,  1997).  As  a  result  of 
this  causes  of  this  mishap  being  was  solely  attributable  to  human  (aircrew)  error,  senior 
Naval  Leadership  established  a  Human  Factors  Quality  Management  Board  (HFQMB) 
with  the  objective  of  reducing  human  error  involvement  in  Naval  Aviation  Class  A  flight 
mishaps  by  50  percent  at  the  start  of  fiscal  year  (FY)  2000  (HFQMB,  1997).  Aircrew 
human  error  was  found  to  be  a  contributing  factor  in  60  percent  of  Class  A  FMs  and, 
consequently,  was  the  initial  focus  of  the  HFQMB.  Although  Naval  Aviation  had  its 
lowest  Class  A  FM  rate  in  FY  1999,  the  HFQMB ’s  goal  of  reducing  human  error  related 
mishaps  by  50  percent  was  not  achieved  (Naval  Safety  Center,  1999).  Thus,  the  scope  of 
the  HFQMB  was  expanded  to  include  the  reduction  of  human  error  in  maintenance 
related  aviation  mishaps.  Maintenance  human  error  contributed  to  about  20  percent  of 
the  Class  A  FM  rate  (Naval  Safety  Center,  1999).  This  thesis  contributes  to  this  endeavor 
by  developing  an  information  management  system  to  facilitate  the  characterization  and 
analysis  of  human  error  in  Naval  Aviation  maintenance  related  mishaps. 
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Fiscal  Year 

Figure  Is  Naval  Aviation  Class  A  Flight  Mishap  Rates  for  FY 1950-1999 
(School  of  Aviation  Safety,  1999) 

The  HFQMB’s  (1997)  strategy  to  achieve  its  objective  consists  of  a  three-pronged 

approach:  (1)  Mishap  Data  Analysis  (MDA),  (2)  Organizational  Benchmarking  (OB), 

and  (3)  Command  Safety  Assessment  (CSA).  MDA  identifies  human  factors  issues  in 

past  Class  A  FMs.  Target  areas  were  prioritized  for  intervention  based  upon  the  presence 

of  prevailing  human  errors.  The  Human  Factors  Analysis  and  Classification  System 

(HFACS)  of  identifying  causal  factors  through  examination  of  past  mishaps  was 

developed  to  achieve  this  end.  OB  is  the  second  approach  used  to  identify  the  best 

practices  and  procedures  in  other  aviation  organizations,  both  military  and  civilian.  For 

example,  the  US  Army  attributed  its  reduced  Class  A  FM  rate  to  the  use  of  Risk 

Management  by  its  aircrew.  Consequently,  Operational  Risk  Management  (ORM)  was 

adopted  by  Naval  Aviation  and  established  as  a  part  of  doctrine  (Department  of  the  Navy- 

-DON,  1997).  CSA  was  developed  to  assess  a  command’s  safety  climate  from  an  aircrew 

perspective  (Ciavarelli  &  Figlock,  1997).  It  solicits  opinions/attitudes  about 

organizational  safety  processes  and  command  climate.  CSAs  may  eventually  show  that 

2 


both  organizational  and  supervisory  issues  impact  flight  safety  (Nutwell  &  Sherman, 
1997). 

In  January  1999,  the  Vice  Chief  of  Naval  Operations  revised  the  goal  to  reduce 
human  error  in  Naval  Aviation  Class  A  FMs  by  50  percent  by  the  end  of  FY2000 
(Personal  communication  between  T.  Meyers  and  B.  Goodrum,  1999).  Presently  one  of 
every  five  Class  A  FMs  contain  maintenance  error.  This  compelled  the  HFQMB  to 
expand  its  focus  to  include  maintenance  related  mishaps  using  the  same  three-prong 
approach  as  used  for  aircrew  error  analysis. 

(2)  Maintenance  Mishap  Data  Analysis  (MDA) 

The  analytic  framework  for  examining  aircrew  and  supervisory  error  in  Class  A 
FMs  was  HFACS.  HFACS  is  a  taxonomy  which  falls  in  line  with  the  Naval  Aviation 
Safety  Program’s  notion  of  multiple  causal  factors,  the  idea  of  sequential  events  leading 
to  an  event,  and  several  established  human  factors  theories.  HFACS  was  adjusted  and 
adopted  to  cover  maintenance  operations,  and  the  extension  was  successfully  used  to 
examine  major  mishaps  (Schmidt,  Schmorrow,  &  Hardee,  1997),  minor  mishaps 
(Schmidt,  Schmorrow,  &  Figlock,  2000),  and  incidents/injury  (Schmidt,  Figlock,  & 
Teeters,  1999)  data.  The  Navy  has  included  an  adjusted  version  of  the  Maintenance 
Extension  of  HFACS  (HFACS-ME)  for  inclusion  in  the  upcoming  revision  of  the  Naval 
Aviation  Safety  Program  Instruction  (DON,  2000). 
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B.  PROBLEM  STATEMENT 

In  order  to  continue  to  reduce  its  Class  A  FM  rate.  Naval  Aviation  leadership 
must  understand  that  all  mishaps  are  not  caused  solely  by  aircrew  error.  The  analysis  of 
maintenance  related  mishaps  offers  an  increased  opportunity  to  reduce  target  mishaps 
and  enhance  readiness.  The  HFACS-ME  taxonomy  has  been  adapted  to  classify  causal 
factors  that  contribute  to  maintenance  mishaps.  A  modem  database  tool  is  essential  in 
more  effectively  addressing  and  identifying  patterns  of  human  error  using  HFACS-ME. 
However,  there  is  no  such  tool  available  today.  The  target  audience  for  such  a  tool  would 
include  safety  personnel  (e.g.,  data  entry  and  retrieval  by  unit  safety  officers,  other  safety 
and  training  personnel,  maintenance  officers,  maintenance  supervisors),  mishap 
investigators-for  data  retrieval  (e.g..  Aircraft  Mishap  Board  members,  squadron  safety 
officers),  and  analysts  (e.g.,  from  the  Naval  Safety  Center,  the  command’s  safety  officer 
or  one  from  its  higher  headquarters). 

This  thesis  investigates  the  following  questions: 

1 .  How  could  human  errors  in  maintenance  related  Naval  Aviation 
mishaps  be  effectively  collected,  cataloged,  and  collated  in  an  information  system? 

2.  How  could  customers  query  and  use  this  maintenance  error  information 
in  order  to  identify  problem  areas  and  trends? 

3.  How  would  customers  in  the  fleet  effectively  and  efficiently  access 
maintenance  error  information  in  Naval  Aviation  mishaps? 
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C.  PURPOSE 


The  intent  of  this  study  is  to  develop  and  evaluate  a  safety  information 
management  system  that  will  facilitate  data  collection,  organization,  query,  analysis,  and 
reporting  of  maintenance  personnel  errors  that  contribute  to  Naval  Aviation  mishaps, 
equipment  damage,  and  personnel  injury  using  the  HFACS-ME  taxonomy  as  its  basis. 

Drawing  upon  several  theoretical  approaches  to  examine  mishaps  involving 
human  error,  including  Heinrich’s  “Domino”  Theory,  Edwards’  “SHEL”  Model,  and 
Reason’s  “Swiss  Cheese”  Model,  helps  to  identify  not  only  the  unsafe  actions  causing  a 
mishap,  but  latent  conditions  which  set  the  stage  for  mishaps  to  occur.  HFACS-ME  is  a 
composite  derivative  of  these  three  taxonomies.  It  classifies  and  analyzes  the  presence  of 
human  error  in  maintenance  operations  leading  to  major  mishaps,  accidents  of  lesser 
severity,  incidents,  and  maintenance  related  personal  injury  cases.  However,  working 
with  a  large  database  by  hand  or  spreadsheet  is  very  labor  intensive.  Given  the  capability 
of  current  relevant  database  tools,  an  improved  information  management  system  will 
bring  HFACS-ME  to  the  next  level  by  improving  access  and  analysis  of  safety  data. 
Though  there  is  no  generally  accepted  method  of  accident  investigation  (Benner,  1975), 
standardized  aircraft  accident  investigation  procedures  have  been  adopted  by  civilian  and 
military  agencies  throughout  the  world  (Diehl,  1991). 

The  result  of  this  study  leads  to  a  development  of  a  tool  that:  (1)  captures 
maintenance  error  associated  with  maintenance  related  incidents;  (2)  facilitates  the 
identification  of  common  maintenance  errors  and  associated  trends;  and  (3)  supports 
understanding  of  how  to  identify  human  errors  in  the  future. 
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D.  SCOPE  AND  LIMITATIONS 


Fleet  personnel,  primarily  Aviation  Safety  Officers,  will  test  the  prototype 
Maintenance  Extension  Information  Management  System  (MEIMS)  tool  (hereafter 
referred  to  as  the  prototype).  The  prototype  to  be  developed  is  to  be  used  by  Naval 
Aviation  squadrons,  but  may  have  some  crossover  use  by  other  military  branches  and 
civilian  airlines.  Only  maintenance  related  mishaps  caused  by  human  error  will  be 
considered.  No  material  failure  factors  or  maintenance  related  hazard  reports  or 
personnel  injuries  not  related  to  a  mishap  are  to  be  included. 

E.  DEFINITIONS 

This  study  uses  the  following  definitions: 

Aircraft  Mishap  Board.  Group  of  officers  appointed  to  investigate  and  report  on 
an  aviation  mishap  (DON,  1991). 

Aviation  mishap  rate.  Number  of  aviation  mishaps  per  100,000  flight  hours 
(DON,  1991). 

Aviation  Safety  Officer.  Principal  advisor  to  Naval  Aviation  squadron 
commanding  officers  on  all  aviation  safety  matters  (DON,  1991) 

F-14  Tomcat.  US  Navy  aircraft.  Two  aircrew,  two  engines,  swing-wing, 
supersonic  fighter  with  air-to-air,  air-to-ground,  and  reconnaissance  capability  (Rowe  & 
Morrison,  1973). 

Fleet  Logistics  Support  Wing.  US  Navy  reserve  air  wing  comprised  of  transport 
aircraft  (Schmidt,  Schmorrow,  &  Teeters,  1999). 
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HFACS:  Human  Factors  Analysis  and  Classification  System.  System  designed 
to  help  analyze  Naval  Aviation  mishaps  focusing  on  aircrew  error  (Shappell  & 
Wiegmann,  1997). 

HFACS-ME:  Human  Factors  Analysis  and  Classification  System-Maintenance 
Extension.  HFACS  adaptation  to  classify  causal  factors  that  contribute  to  maintenance 
mishaps  (Schmidt,  1996). 

HFQMB:  Human  Factors  Quality  Management  Board.  Established  by  Naval 
Aviation  senior  leadership  to  reduce  human  error  involvement  in  Naval  Aviation  Class  A 
flight  mishaps  (HFQMB,  1997). 

MEIMS:  Maintenance  Error  Information  Management  System.  Prototype  tool 
developed  for  this  thesis. 

Mishap.  A  naval  mishap  is  an  unplanned  event  or  series  of  events  directly 
involving  naval  aircraft,  which  result  in  $10,000  or  greater  cumulative  damage  to  naval 
aircraft,  other  aircraft,  property,  or  personnel  injury  (DON,  1991). 

Mishap  Categories.  Naval  aircraft  mishap  categories  are  defined  below  (DON, 

1991): 

Flight  Mishap  (FM).  Those  mishaps  in  which  there  was  $10,000  or  greater 
DOD  aircraft  damage  or  loss  of  a  DOD  aircraft,  and  intent  for  flight  for  DOD 
aircraft  existed  at  the  time  of  the  mishap.  Other  property  damage,  injury,  or  death 
may  or  may  not  have  occurred. 

Flight  Related  Mishap  (FRM).  Those  mishaps  in  which  there  was  less  than 
$10,000  DOD  aircraft  damage,  and  intent  for  flight  (for  DOD  aircraft)  existed  at 
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the  time  of  the  mishap,  and  $10,000  or  more  total  damage  or  a  defined  injury  or 
death  occurred. 

Aircraft  Ground  Mishap  (AGM).  Those  mishaps  in  which  no  intent  for 
flight  existed  at  the  time  of  the  mishap  and  DOD  aircraft  loss,  or  $10,000  or  more 
aircraft  damage,  and/or  property  damage,  or  a  defined  injury  or  death  occurred. 
Mishap  Severity  Class.  Mishap  severity  classes  are  based  on  personnel  injury  and 
property  damage  (DON,  1991): 

Class  A.  A  mishap  in  which  the  total  cost  of  property  damage  (including 
all  aircraft  damage)  is  $1,000,000  or  greater;  or  a  naval  aircraft  is  destroyed  or 
missing;  or  any  fatality  or  permanent  total  disability  occurs  with  direct 
involvement  of  naval  aircraft. 

Class  B.  A  mishap  in  which  the  total  cost  of  property  damage  (including 
all  aircraft  damage)  is  $200,000  or  more,  but  less  than  $1,000,000  and/or  a 
permanent  partial  disability,  and/or  the  hospitalization  of  five  or  more  personnel. 

Class  C.  A  mishap  in  which  the  total  cost  of  property  damage  (including 
all  aircraft  damage)  is  $10,000  or  more  but  less  then  $200,000  and/or  injury 
results  in  five  or  more  lost  workdays. 

Naval  Aircraft.  Refers  to  US  Navy,  US  Naval  Reserve,  US  Marine  Corps,  and  US 
Marine  Corps  aircraft. 

OPNAVINST  3750.6:  The  Naval  Aviation  Safety  Program.  US  Navy  instruction 
outlining  Naval  Aviation’s  safety  program.  Revision  Q-1991,  revision  R-in  work  (DON, 
1991  &  2000). 
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ORM:  Operational  Risk  Management.  A  decision  making  tool  to  increase 
effectiveness  (and  hence  decrease  accidents)  by  anticipating  hazards,  reducing  the 
potential  for  loss  due  to  these  hazards,  and  thus  increasing  the  probability  of  a  successful 
mission  (DON  1997). 

F.  ORGANIZATION  OF  STUDY 

Chapter  II  contains  a  literature  review  on  the  development  of  a  prototype  to 
identify  human  error  involvement  and  patterns  in  aviation  maintenance  mishaps.  The 
methods  used  in  this  study  are  discussed  in  Chapter  III.  The  results  of  this  study  are 
presented  in  Chapter  IV.  Finally,  Chapter  V  contains  conclusions,  findings,  and 
recommendations. 
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II.  LITERATURE  REVIEW 


A.  OVERVIEW 

The  literature  examined  relates  to  the  development  of  a  prototype  to  identify 
human  error  involvement  and  patterns  in  aviation  maintenance  mishaps.  It  includes 
textbooks,  research  articles,  and  masters  theses  pertaining  to:  (1)  the  collection,  use,  and 
management  of  accident  information,  (2)  human  error  theories,  its  involvement  in 
aviation  mishaps,  and  specifically  maintenance  mishaps,  and  (3)  design  and  usability  of 
an  effective  mishap  database  tool.  Collectively,  these  information  sources  provide  a 
foundation  to  develop  an  effective  and  user  friendly  maintenance  error  analysis  and 
reporting  tool. 

Diehl  (1991),  in  a  three-stage  model  of  accident  investigation  and  prevention, 
focuses  on  human  performance  and  systems  safety  considerations  (see  Figure  2).  The 
first  stage  is  Accident  Generation:  the  identification  of  hazards.  Hazards  have  the 
potential  to  lead  to  an  incident  (near-accident)  or  even  an  accident.  Heinrich  (1941) 
study  of  thousands  of  accidents  determined  that  for  every  major  accident,  there  are 
approximately  30  minor  accidents,  and  300  hazardous  incidents.  This  pyramidal 
relationship  between  hazards,  incidents,  and  accidents  also  applies  to  aviation  safety 
(Diehl,  1991). 
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Figure  2:  Accident  Generation,  Investigation,  and  Prevention  Elements 
(Adapted  from  Diehl,  1989) 

The  second  stage  is  the  Accident  Investigation  Process:  the  collection,  analysis, 
and  review  of  accident  data  and  the  focus  of  this  review.  Accidents  rarely  result  from  a 
single  sudden  event,  but  are  normally  associated  with  a  series  of  events  degrading  the 
performance  of  the  equipment,  crewmen,  or  both,  until  the  accident  is  inevitable  (Nance, 
1986).  Investigating  bodies  have  established  similar  aircraft  accident  investigation 
procedures.  The  fact-finding  phase  takes  place  near  the  scene  of  the  accident  to  establish 
what  happened.  Next,  the  information  analysis  emphasizes  on  describing  what  caused 
the  accident  and  why  it  occurred  (Diehl,  1991).  Part  of  that  analysis  is  based  on 
examining  comparative  data  sources:  information  about  both  the  normal  and  emergency 
performance  of  the  aircraft,  as  well  as  human  capabilities  and  limitations.  Investigators 
are  now  able  to  theorize  as  to  the  causal  factors  of  the  accident  and  its  probable  sequence 
of  events.  Once  the  analysis  is  complete  a  final  report  is  developed  by  board  authorities 
with  the  accepted  findings,  causes,  and  recommendations.  This  phase  is  judgmental  in 
nature. 
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The  final  stage  contains  the  Prevention  Measures  (and  methods)  used  to  avoid 
future  similar  incidents.  There  are  four  categories  of  accident-prevention  measures: 

(1)  eliminating  hazards  and  risks,  (2)  incorporating  safety  features  (3)  providing  warning 
devices,  and  (4)  establishing  procedural  safeguards.  As  one  travels  from  right  to  left 
along  the  bottom  leg  of  Diehl’s  triangle,  the  measures  become  less  expensive,  less 
effective,  and  less  restrictive.  (Diehl,  1991). 

B.  ACCIDENT  INFORMATION 

(1)  Investigation 

Accidents  occur  within  an  organizational/systems  context,  and  understanding  the 
involved  systems  and  operating  environment  can  provide  an  enhanced  framework  for 
investigating  accidents  and  determining  their  causes  (Wagenaar,  Groeneweg,  &  Hudson, 
1994).  During  the  initial  phase  of  an  investigation  retrospective  analysis  of  past  accidents 
can  help  to  focus  on  areas  of  high  risk  and  identify  groups  of  potential  causal  factors 
(McElroy,  1974).  Effective  interventions  can  then  be  identified  and  subsequently 
implemented  to  reduce  accident  occurrence.  However,  the  perceptions  of  accident 
investigators  can  be  skewed  and  thus  diminish  the  effectiveness  of  an  investigation 
(Benner,  1982).  Therefore,  a  systematic  process  for  investigating  and  reporting  accidents 
is  imperative. 

Grimaldi  &  Simonds  (1984)  detailed  a  four-part  process  for  investigations.  The 
first  step  is  to  explore  the  history  of  the  incident  as  far  back  as  is  practical,  including 
activities  occurring  both  during  and  prior  to  the  event.  Second,  the  investigator  must 
collect  as  many  facts  relating  to  the  incident  as  possible  (from  reliable  witnesses).  Next, 
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the  physical  environment  associated  with  the  accident  must  be  examined.  Finally,  the  use 
of  a  defining  guide  listing  common  causal  factors  can  be  used  to  determine  probable 
causal  factors  of  the  incident  itself.  This  process  parallels  aspects  of  that  provided  by 
Diehl  (1991)  in  his  model  of  aviation  accident  investigation. 

Though  there  is  no  generally  accepted  method  of  accident  investigation  (Benner, 
1975),  standardized  aircraft  accident  investigation  procedures  have  been  adopted  by 
civilian  and  military  agencies  throughout  the  world  (Diehl,  1991). 

(2)  Reporting 

Accident  reports  have  generally  centered  on  number  of  episodes  and  observations 
per  unit  time  (Brown,  1990a).  Frequencies  and  rates  alone,  however,  do  not  provide  a 
sound  basis  for  understanding  accidents  (Brown,  1990a).  The  conventional  process  of 
reporting  accidents  by  a  description  followed  by  supporting  documentation  varies  in 
scope,  depth,  quality,  objectivity,  and  contains  inconsistencies  and  varying  levels  of 
completeness  (Edwards,  1981).  In  addition,  the  traditional  reporting  format  does  not 
normally  capture  human  factors  information  (Adams,  Barlow,  &  Hiddlestone,  1981).  To 
increase  the  usability  of  mishap  reports,  the  information  they  contain  must  assist  in  the 
determination  of  cause  and  prevention  of  future  accidents  by  ensuring  collection, 
classification,  and  data  recording  methods  are  accurate  and  reliable.  The  usefulness  of 
the  reports  is  greatly  increased  when  bias  is  removed  and  any  future  potential  (based  on 
frequency  or  severity)  of  occurrence  is  easily  used  (Adams  &  Hartwell,  1977). 

Three  elements  critical  to  the  success  of  an  accident  reporting  system  are 
(Chapanis,  1996):  (1)  properly  trained  investigators,  (2)  a  good  accident  reporting  form, 
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and  (3)  a  centralized  facility  for  dealing  with  reports.  If  a  mishap  reporting  system  is 
unable  to  either  prevent  or  reduce  the  severity  of  future  accidents  (Brown,  1990b) 
subsequent  data  analysis  can  be  problematic  (Pimble  &  O’Toole,  1982).  Analysis  of  data 
from  typical  reporting  systems  has  been  accomplished  through  the  following  process: 

•  collecting  data  on  past  accidents  within  a  population; 

•  dividing  the  sample  into  groups  with  and  without  accidents; 

•  obtaining  measurements  of  individual  characteristics  on  all  participants; 

•  statistically  comparing  the  two  groups;  and 

•  identifying  any  significant  difference  between  the  two  groups,  associating  the 
differential  characteristic  with  accidents. 

Using  these  methods  has  resulted  in  a  more  complete  and  thorough  analysis  effort. 

This  general  style  of  accident  reporting  has  been  used  by  many  studies,  but  its 
methods  have  also  been  concluded  to  be  suspect  (Hale  &  Hale,  1972;  Hansen  1988;  Shaw 
&  Sichel,  1971).  The  symptom  may  not  actually  be  responsible  for  the  accident,  but  may 
be  related  to  another  (correlative)  variable  which  may,  in  fact,  bear  responsibility. 
Recently,  accident  reporting  tools  have  been  advanced  and  are  supporting  more  rigorous 
and  ordered  methods  of  analysis  (Leplat,  1989;  Malaterre,  1990;  Reason,  1990.  The 
ability  of  a  report  to  distinguish  between  causal  and  correlative  variables  determines  its 
utility  (Hill,  Byers,  Rothblum,  &  Booth,  1994). 
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(3)  Data  Management 

Once  data  pertaining  to  an  accident  is  collected,  it  must  be  archived  for  use  in 
accident  prevention  (National  Safety  News,  1975).  Many  methods  of  recording  this 
information  are  in  use,  but  some  fundamental  concepts  are  recognized  including  coding 
the  data  and  the  use  of  computer  databases  to  quickly  and  efficiently  store  and  retrieve 
information.  In  addition,  the  attributes  of  the  data  are  critical  in  ensuring  the  best 
information  is  collected  and  stored  for  future  use. 

The  National  Safety  Council  (National  Safety  News,  1975)  established  a  method 
of  facilitating  the  sorting  and  tabulation  of  accident  particulars.  Numerical  codes  are 
assigned  to  the  different  classifications  in  the  mishap.  Therefore,  the  specific  case  is  read 
once  when  its  facts  are  assigned  code  numbers.  Concentrating  on  one  phase  of  the 
accident  problem  at  a  time  is  a  more  effective  way  to  reduce  incidents  than  to  deliberate 
on  the  mishap  as  a  whole.  Merely  obtaining  the  information  will  not  prevent  recurrence 
of  the  accidents.  The  conditions  contributing  to  the  incident  must  be  corrected. 
Subsequent  sorting  of  these  facts  by  category  can  then  be  completed  quickly  by  simply 
referencing  the  code  numbers. 

The  Swedish  Information  System  (ISA)  on  Occupational  Accidents  and  Diseases 
was  developed  in  1979  to  improve  the  work  environment  through  increasing  knowledge 
about  risks  (Andersson  &  Lagerlof,  1983).  The  ISA’s  goal  is  accomplished  through  the 
collection  of  information  in  four  areas:  (1)  knowledge  about  risks,  (2)  knowledge  about 
preventive  actions,  (3)  cost-benefit  analysis,  and  (4)  a  “will”  to  change  (see  Figure  3).  To 
identify  a  risk,  the  experience  of  one  accident  of  a  specific  type  is  viewed  as  enough,  and 
consequently  accident  prevention  has  traditionally  been  very  case-related.  However, 
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experience  alone  is  not  satisfactory  for  identifying  and  evaluating  risks  since  minor 
incidents  are  frequently  given  less  attention  and  major  ones  occur  less  frequently.  The 
ISA  system  provides  reports  based  on  its  database:  periodic  statistics,  focused  statistics, 
figures  for  particular  accidents,  and  frequency  of  accidents. 


Figure  3:  Swedish  Information  System  on  Occupational  Accidents  and 
Diseases  (ISA)  (Andersson  &  Lagerlof,  1983) 

It  is  common  to  have  a  system  that  only  reports  incidents  resulting  in  an  accident 
(Grimaldi  &  Simonds,  1984).  However,  it  is  important  to  recognize  “near-miss”  cases  in 
order  to  identify  potential  conditions  or  practices  that  are  accident  producing  types  and 
prevent  their  future  occurrence.  Essentially  the  same  form  could  be  used  in  both  accident 
and  near-miss  incidents. 

Setting  up  a  computer  analysis  can  reduce  man-hours  involved  in  reviewing 
mishap  histories  (Kuhlman,  1977).  To  set  up  an  effective  program,  the  available 
information  needs  to  be  organized  and  tabulated  into  categories.  This  information  then 
can  be  presented  in  a  format  (report)  to  the  user  who  can  decide  how  to  use  it  to  analyze 
past  occurrences. 
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(4)  Prevention 

Interest  in  accident  prevention  did  not  begin  until  the  beginning  of  the  20th 
century  when  employers  realized  that  it  was  less  expensive  to  prevent  accidents  than  to 
pay  for  their  consequences  (Petersen,  1978).  Organizations  confronted  with  the  challenge 
of  how  best  to  protect  themselves  and  their  employees  from  accidents  have  two  options, 
namely,  insurance  and  accident  prevention  programs  (Pate-Comell,  1996),  and 
organizations  typically  employ  both  options  (Kanis  &  Weegels,  1990). 

Accident  prevention  was  initially  based  on  a  widely  held  notion  that  people 
committing  unsafe  acts,  not  their  working  conditions,  were  to  blame  for  most  accidents 
(Heinrich,  1959).  This  thinking  fostered  a  preoccupation  with  assigning  blame  to  people; 
a  practice  which  hindered  the  development  of  systematic  accident  prevention  well  into 
the  later  half  of  this  century  (Manuele,  1981).  Narrowly  focusing  on  people  and  not  on 
the  environment  in  which  they  operate,  tended  to  obscure  a  subset  of  associated  causal 
factors.  This  is  particularly  true  with  systems  that  chronically  expose  individuals  to 
hazards  (Schmidt,  1987).  Although  there  have  been  substantial  advances  in  accident 
prevention  in  recent  decades,  the  practice  of  blaming  individuals  for  the  accident,  rather 
than  the  conditions  associated  with  it,  persists.  This  practice  must  be  overcome  and 
accidents  must  be  analyzed  in  terms  of  the  systems  in  which  they  occur. 

The  most  effective  accident  prevention  strategies  employ  systems  engineering 
(Hawkins,  1993).  The  systems  engineering  approach  was  developed  in  the  1950s  as  part 
of  the  United  States  military’s  large-scale  weapons  programs.  Systems  engineering 
transforms  operational  needs  into  a  description  of  system  parameters  and  integrates  them 
to  optimize  overall  system  effectiveness  (Edwards,  1988).  In  addition,  it  focuses  the  level 
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of  analysis  on  the  smallest  identifiable  system  components  and  how  these  components 
interact  (Bird,  1980).  The  strategy  of  focusing  on  the  system  through  the  development  of 
well-defined  system  components  exposes  information  that  would  have  remained 
unknown  without  a  system-level  evaluation  (Miller,  1988). 

Systems  engineering  pays  attention  to  the  strengths  and  limitations  of  the  human 
operator  as  an  integral  part  of  the  system  (Heinrich,  Petersen,  &  Roos,  1980).  The 
literature  suggests  that  80-90  percent  of  accidents  are  attributable  to  human  error 
(Heinrich,  Petersen,  &  Roos,  1980;  Hale  &  Glendon,  1987;  School  of  Aviation  Safety, 
2000).  Therefore,  evaluating  human  factors  associated  with  accidents  can  contribute  to 
the  understanding  of  systems  and  how  they  fail. 

Operational  Risk  Management  (ORM)  is  another  tool  used  by  the  armed  forces  to 
decrease  aviation  accident  rates.  It  is  a  decision  making  tool  used  by  personnel  at  all 
levels  to  increase  effectiveness  (and  hence  decrease  accidents)  by  anticipating  hazards, 
reducing  the  potential  for  loss  due  to  these  hazards,  and  thus  increasing  the  probability  of 
a  successful  mission  (DON,  1997).  The  aviation  arm  of  the  United  States  Army  achieved 
record  low  accident  rates  in  1995  and  1996  attributing  a  large  portion  of  their  success  to 
the  use  of  ORM  (Department  of  the  Army,  2000).  The  remaining  military  services 
institutionalized  ORM  in  1997  in  attempt  to  lower  their  own  rates  (School  of  Aviation 
Safety,  2000).  ORM  emphasizes  identifying  hazards  and  reducing  their  associated  risk  to 
an  acceptable  level  through  the  use  of  control  measures  (DON,  1997).  It  is  especially 
effective  in  identifying  and  analyzing  human  factor  hazards  as  well. 
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C.  HUMAN  ERROR 


Knowledge  derived  from  the  analysis  of  human  errors  can  greatly  improve  safety. 
There  are  numerous  theoretical  approaches  to  examine  mishaps  involving  human  error 
(Goetsch,  1996).  Some  methods  have  their  basis  in  industrial  safety,  while  others  are 
viewed  from  a  more  complex  systems  perspective,  with  an  emphasis  on  human  factors 
and  operator  error.  Table  1  outlines  some  of  the  more  well-known  approaches. 


Table  1:  Theoretical  Approaches  to  Defining  Accident  Processes  (Schmidt,  1998) 


Source 

Model 

Approach 

Industrial  Safety 

Heinrich’s  Domino  Theory 

Linear 

Systems  Safety 

Edwards’  SHEL  Model 

Interface 

Human  Factors 

Reason’s  Swiss  Cheese  Model 

Vertical 

(1)  Heinrich’s  “Domino”  Theory 

The  original  accident  causation  theory  is  considered  to  be  Heinrich’s  “Domino” 
Theory  (Goetsch,  1996).  Heinrich  believed  accidents  could  be  viewed  as  a  linear  five 
step  sequence  of  related  factors  (chain  of  events)  that  lead  to  an  actual  mishap  (Bird, 
1980).  The  two  central  principles  of  the  Domino  Theory  are  (Goetsch,  1996):  (1) 
accidents  are  caused  by  the  actions  of  the  preceding  factors,  and  (2)  removal  of  the 
middle  factor  (unsafe  act  or  condition)  will  negate  the  actions  of  the  preceding  factors 
and  thus  prevent  accidents  and  injuries  (see  Figure  4). 
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Figure  4:  Domino  Theory  (School  of  Aviation  Safety,  1998) 


The  following  is  a  characterization  of  each  domino  (Bird,  1980): 

•  Lack  of  Control  by  Management  (lack  of  supervision).  The  professional 
manager  has  four  control  functions:  planning,  organizing,  leading,  and 
controlling.  Managers  at  all  levels  and  all  activities  must  perform  these 
functions  to  ensure  proper  completion  of  work. 

•  Basic  Cause(s)  of  incident— Origin(s).  A  lack  of  management  control  (domino 
one)  allows  certain  basic  causes  of  incidents  to  exist.  These  causes  are 
classified  into  two  separate  categories: 

o  Personal  Factors:  denoted  by  a  lack  of  knowledge  or  skill,  improper 
motivation,  physical  or  mental  problems.  Personal  factors  explain 
why  people  engage  in  substandard  practices, 
o  Job  Factors:  denoted  by  inadequate  work  standards,  inadequate  design 
or  maintenance,  inadequate  purchasing  standards,  normal  wear  and 
tear,  abnormal  usage.  Job  factors  explain  why  substandard  conditions 
exist. 
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•  Immediate  Cause(s)— Symptoms  (unsafe  act/condition).  If  basic  causes  of 
incidents  exist,  the  opportunity  for  actual  substandard  practices  and  conditions 
(errors)  also  exists.  A  substandard  practice  or  condition  is  a  deviation  from  an 
accepted  standard  or  practice. 

•  Incident-Contact.  If  substandard  practices  and  conditions  exist,  an  incident 
may  occur  that  may  or  may  not  result  in  a  loss.  Of  note,  every  incident  that 
occurs  provides  an  opportunity  to  collect  data  that  could  prevent  a  future 
occurrence. 

•  Accident:  Loss  of  People  or  Property.  Once  an  incident  has  occurred  it  can 
result  in  a  loss  of  personnel  or  property. 

Each  step  causes  the  next  to  occur,  as  would  a  series  of  falling  dominos.  If  factors  from 
any  of  the  first  three  dominos  are  removed,  the  accident  will  effectively  be  prevented. 

(2)  Edwards’  “SHEL”  Model 

The  “SHEL”  Model  (Edwards,  1988)  was  developed  in  the  early  1970s  to  provide 
a  more  effective  means  to  evaluate  human-machine  systems  failures.  The  model 
identifies  and  classifies  four  dimensions  in  evaluating  human-machine  systems  failures: 
Software,  Hardware,  Environment,  and  Liveware. 

•  Software  (S):  rules,  regulations,  laws,  orders,  standard  operating  procedures, 
customs,  practices,  and  habits  that  govern  the  manner  in  which  the  system 
operates  and  in  which  the  information  within  it  is  organized;  typically,  a 
collection  of  documents. 
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•  Hardware  (H):  buildings,  vehicles,  equipment,  and  materials  of  which  the 
system  is  comprised. 

•  Environmental  conditions  (E):  operating  setting  (physical,  economic,  political 
and  social  factors)  of  the  software,  hardware,  and  liveware  operate. 

•  Liveware  (L):  people  involved  with  the  system. 

Thus  the  SHEL  Model  is  comprised  of  these  system  dimensions  and  the 
relationships  between  them  (see  Figure  5). 

Software 
Hardware 
Environment 
liveware 
Liveware 

Figure  5:  SHEL  Model  of  System  Design  (Hawkins,  1993) 

The  SHEL  Model  assumes  that  a  failure  in  the  system  will  occur  when  one  of  the 
dimensions  or  the  connection  between  them  fails  (Edwards,  1988).  People  are  rarely  the 
only  cause  of  a  mishap.  They  are,  in  fact,  caused  by  the  interaction  of  many  factors 
(Shappell  &  Wiegmann,  1997).  The  SHEL  Model  is  a  significant  change  from  the 
previously  held  idea  that  mishaps  have  single  cause  factors  (Edwards,  1981).  The  SHEL 
Model  describes  systems,  identifies  areas  for  concern  in  a  system,  and  provides  a 
framework  for  accident  investigation. 
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(3)  Reason’s  “Swiss  Cheese”  Model 

The  Swiss  Cheese  model  (Reason,  1 990)  of  accident  causation  is  another  widely 
accepted  perspective.  Reason  took  a  human  factors  approach  to  view  the  vertical 
association  of  a  group  of  factors  that  lead  to  an  eventual  accident.  His  model 
differentiates  between  two  error  types:  (1)  active  failures-the  actions  (or  inactions)  of 
operators  that  are  believed  to  have  caused  the  accident,  and  (2)  latent  conditions- 
situations  primarily  caused  by  management  decisions  or  actions  whose  repercussions  may 
only  become  apparent  when  they  are  triggered  by  other  mitigating  factors.  The 
conjunction  between  context  and  acts,  when  taken  together,  are  latent  conditions.  Latent 
conditions  set  the  groundwork  for  an  accident  while  active  failures  are  the  final  catalyst 
for  the  mishap  to  occur.  Safeguards  in  a  system  can  prevent  latent  conditions  from  taking 
effect  by  reducing  the  probability  for  the  commission  of  an  active  failure.  Thus  Reason’s 
model  seen  as  Swiss  cheese  slices  lined  in  a  row,  with  each  vertical  slice  representing  a 
defense  layer  and  each  hole  representing  an  active  failure  or  latent  condition  in  the 
defense.  An  accident  will  occur  when  the  holes  are  aligned  (see  Figure  6). 


Figure  6:  Reason’s  Swiss  Cheese  Model  (Schmidt  &  Lawson,  2000) 
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Reason  s  (1997)  model  is  not  static,  but  dynamic,  with  each  defensive  layer 
moving  according  to  the  characteristics  of  the  situation  (Reason,  1997).  An  event  may 
occur  in  one  of  three  levels:  (1)  person-unsafe  acts,  (2)  workplace-error  provoking 
conditions,  and  (3)  organization-error  establishing  conditions.  The  starting  point  for  an 
accident  occurs  with  organizational  factors;  strategic  decisions  and  associated  processes 
(resource  allocation,  budgeting,  forecasting,  planning)  are  initiated.  These 
organizationally  established  processes  are  shaped  and  influenced  by  a  corporate  culture 
being  distributed  throughout  the  organization  to  individual  workplaces.  Corporate 
processes  evidence  themselves  as  inadequate  staffing,  time  pressures,  equipment, 
training,  and  working  conditions.  These  factors,  combined  with  the  natural  proclivity  to 
commit  errors  and/or  violations  results  in  unsafe  acts.  Very  few  of  these  acts  actually 
create  holes  in  the  defense  layers  en  route  to  becoming  an  accident  (Reason,  1997). 

(4)  Human  Factors  Analysis  &  Classification  System  (HFACS) 

A  restructuring  and  expansion  of  the  Swiss  Cheese  Model  evolved  into  HFACS 
which  was  specifically  designed  to  help  analyze  Naval  Aviation  mishaps  (Shappell  & 
Wiegmann,  1997).  HFACS  focuses  on  aircrew  error  and  also  incorporates  features  of 
Heinrich  s  Domino  Theory  and  Edwards’  SHEL  Model.  The  resulting  taxonomy  of 
unsafe  operations  identifies  both  active  failures  and  latent  conditions  within  four 
categories  (DON,  2000):  (1)  unsafe  acts;  (2)  pre-conditions  for  unsafe  acts;  (3)  unsafe 
supervision,  and  (4)  organizational  influences.  This  classification  can  then  be,  and,  in 
fact,  has  been,  used  to  target  the  most  appropriate  intervention  (see  Figure  7).  The  Naval 
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Safety  Center  has  adopted  the  use  of  the  HFACS  model  for  analysis  of  aircrew  error  in 
Naval  Aviation  mishaps. 


Figure  7:  Levels  of  Human-Component  Failure  (Schmidt,  1998) 

The  following  is  a  brief  description  of  the  HFACS  taxonomy  (DON,  2000). 
Unsafe  acts  take  two  forms:  (1)  errors  and  (2)  violations.  Errors  are  found  in  most 
mishaps  due  to  the  facts  that  human  beings  by  their  nature  make  mistakes  and  are  often 
the  last  flaw  before  the  mishap  occurs.  There  are  three  basic  error  types:  (1)  decision,  (2) 
perceptual,  and  (3)  skill-based.  Violations,  on  the  other  hand,  are  the  willful  disregard  for 
the  rules  and  are  not  seen  in  as  many  mishaps.  They  are  also  further  categorized  into 
routine  and  exceptional  violation  categories.  Pre-conditions  for  unsafe  acts  set  the  table 
for  the  unsafe  act  to  occur.  Its  two  major  subdivisions  are  (1)  substandard  conditions  of 
operators  and  (2)  substandard  practices  of  operators.  Substandard  conditions  included 
adverse  mental  and  physiological  states  and  physical/mental  limitations.  Substandard 
practices  include  crew  resource  management  and  personal  readiness.  Failures  associated 
with  unsafe  supervision  are  divided  into  four  areas:  (1)  inadequate  supervision,  (2) 
planned  inappropriate  operations,  (3)  failure  to  correct  a  known  problem,  and  (4) 
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supervisory  violations.  Upper-level  management  failures,  or  organizational  influences, 
directly  effect  not  only  supervisory  practices,  but  operator  conditions  and  actions.  These 
latent  failures  can  be  traced  to  issues  dealing  with  resource  management,  organizational 
climate,  and  operational  processes. 


(5)  Human  Error  in  Maintenance  Mishaps 

Maintenance  level  human  factors  in  aircraft  mishaps  can  be  categorized  similarly 
to  aircrew  level  human  factors  (DON,  2000).  The  Maintenance  Extension  (ME)  of  the 
HFACS  taxonomy  was  adapted  to  classify  causal  factors  that  contribute  to  maintenance 
mishaps  (Schmidt,  1996).  It  contains  four  human  error  categories:  (1)  Supervisory 
Conditions;  (2)  Working  Conditions;  (3)  Maintainer  Conditions;  and  (4)  Maintainer  Acts 
(see  Figure  8).  These  categories  provide  for  multiple  causations,  a  chain  of  inter-related 
events,  and  observation  between  the  link  between  components  providing  for  a  combined 
approach  to  study  human  error  and  its  causes. 
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Supervisory,  Working,  and  Maintainer  Conditions  are  latent  conditions  that  can 
impact  the  performance  of  a  maintainer  (Schmidt,  Schmorrow,  &  Hardee,  1997).  This 
may  contribute  to  maintainer  act,  an  active  failure,  leading  directly  to  a  maintenance 
related  mishap  (MRM),  maintenance  condition,  or  personal  injury.  Thus,  the  HFACS- 
ME  categories  enable  a  safety  analyst  to  identify  failures  at  each  of  the  four  levels 
historically  related  to  accidents  (Schmidt,  Schmorrow,  &  Hardee,  1997).  The  working 
conditions  of  a  maintainer,  as  compared  to  those  of  the  aircrew,  will  often  play  a  more 
significant  role  in  errors  observed  during  maintenance  evolutions  (DON,  2000). 
Maintenance  conditions  have  the  potential  to  become  a  latent  condition  with  which  the 
aircrew  would  have  to  accommodate  in  flight  and  can  also  directly  lead  to  mishap  or 
injury  through  no  fault  of  the  aircrew.  The  three  orders  of  maintenance  error:  first, 
second,  and  third  order,  reflect  a  decomposition  of  the  error  type  from  a  macro  to  a  micro 
perspective  (see  Table  2). 

The  following  describe  the  categories  of  the  original  HFACS-ME  taxonomy. 
Supervisory  Conditions  may  contribute  to  an  active  failure  due  to  either  unforeseen  or 
squadron  errors.  Maintainer  Conditions  that  can  contribute  to  an  active  failure  include 
medical,  crew  resource  management,  and  personal  readiness.  (Schmidt,  1998) 

Working  Conditions  include  the  physical  environment  in  which  the  maintainer 
works  and  the  tools  they  use  in  the  course  of  their  work.  Circumstances  that  can 
contribute  to  an  active  failure  include  poor  environmental  factors  (lighting,  weather, 
environmental  hazards),  inadequate  equipment  (damaged,  unavailable,  uncertified),  and 
uncomfortable  workspaces  (confining,  obstructed,  inaccessible).  (Schmidt,  1998) 
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Table  2:  HFACS  Maintenance  Extension  Categories  (DON,  2000) 


First  Order 

Second  Order 

Third  Order 

Supervisory  Conditions 

Organizational 

Hazardous  Operations 
Inadequate  Documentation 
Inadequate  Design 
Inadequate  Processes 
Inadequate  Resources 

Squadron 

Inadequate  Supervision 
Inappropriate  Operations 
Uncorrected  Problem 
Supervisory  Misconduct 

Maintainer  Conditions 

Medical 

Unsafe  Mental  State 

Unsafe  Physical  State 

Unsafe  Limitation 

Crew  Coordination 

Inadequate  Communication 
Inadequate  Assertiveness 
Inadequate  Adaptability/Flexibility 

Readiness 

Inadequate  Training/Preparation 
Inadequate 

Certification/Qualification 
Personnel  Readiness  Infringement 

Working  Conditions 

Environment 

Inadequate  Lighting/Light 
Unsafe  Weather/Exposure 
Unsafe  Environmental  Hazards 

Equipment 

Damaged 

Unavailable 

Dated/Uncertified 

Workspace 

Confining 

Obstructed 

Inaccessible 

Maintainer  Acts 

Error 

Attention 

Memory 

Knowledge/Rule  Based 

Skill  Based 

Judgment/Decision  Making 

Violation 

Routine 

Infraction 

Flagrant 

Sabotage 
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Errors  and  violations  are  active  failures  in  the  form  of  Maintainer  Acts.  Active 
Failures  can  either  directly  cause  damage  and  injury,  or  lead  to  a  latent  Maintenance 
Condition.  Errors  is  substandard  performance  due  to  inattention,  poor  workmanship,  and 
complacency.  Violations  are  intended  actions  including  both  the  routine  or  exceptional 
variety.  Routine  violations  are  consistent  departures  from  rules  and  regulations  condoned 
by  management.  Thus  routine  violations  are  considered  to  be  acceptable  departures  from 
rules  and  regulations.  Exceptional  violations  are  substandard  practices  and  actions  not 
condoned  by  management.  (Schmidt,  1998). 

HFACS-ME  was  effective  in  capturing  the  nature  of  and  relationships  among 
active  failures  and  latent  conditions  present  in  63  Class  A  (hull  loss  or  fatality)  Naval 
Aviation  maintenance  mishaps  (Schmidt,  Schmorrow,  &  Hardee,  1997),  470  reportable 
(over  $10,000  damage  or  permanent/partial  disability)  Naval  Aviation  maintenance 
mishaps  (Schmidt,  Schmorrow,  &  Figlock,  2000),  124  incidents  (Mishap  Reports,  Hazard 
Reports,  and  Injury  Reports)  for  Fleet  Logistics  Support  Wing  maintenance  operations 
(Schmidt,  Figlock,  &  Teeters,  1999),  and  15  select  National  Transportation  Safety  Board 
(NTSB)  major  (hull  loss  or  fatality)  maintenance  accidents.  The  insight  into  latent 
conditions  and  active  failures  provides  a  solid  perspective  for  trend  analysis,  investigation 
prioritization,  and  control  development. 

(6)  Maintenance  Error  Issues 

Marx  (1998)  stated  that  human  error  has  not  been  served  well  by  conventional 
accident  investigation  methods.  These  processes  normally  end  once  human  error  is 
identified  without  trying  to  understand  why  it  occurred.  This  problem  has  been  attributed 
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to  several  factors  (Schmidt,  1998):  (1)  reporting  criteria,  (2)  investigator  biases,  (3)  report 
scope,  depth,  and  quality,  (4)  reporting  system  design,  and  (5)  database  construction.  By 
focusing  on  a  human  factors  oriented  investigation  and  reporting  process,  we  can 
understand  why  people  make  certain  mistakes. 

The  prevention  of  accidents  is  critically  linked  to  a  sufficient  investigation  of 
human  factors  (Harle,  1994).  Such  investigation  methods  must  be  properly  designed, 
implemented  and  supported.  Past  attempts  at  this  have  generated  more  superficial 
information  than  substance  (Zotov,  1996)  and  have  failed  to  properly  consider  the  human 
element  in  the  system  (Bruggink,  1996).  Human  factors  based  investigation  methods  are 
considered  by  aviation  industry  personnel  to  be  a  better  form  of  inquiry;  however,  they 
are  not  being  widely  used  (Schmidt,  1998).  Reason’s  model  established  a  conceptual 
framework  of  human  error  that  has  been  widely  accepted  throughout  government, 
military,  and  commercial  sectors.  Despite  this  acceptance,  his  model  does  not  completely 
define  the  forerunners  to  accidents  (Marx,  1998). 

(7)  Maintenance  Error  Decision  Aid  (MED A) 

Boeing  Aircraft  Corporation  developed  an  event-driven  tool  to  reduce 
maintenance  related  accidents  by  assisting  investigators  in  the  identification  of  accident 
contributing  factors  and  recommendations  for  Correction-Maintenance  Error  Decision 
Aid  or  “MED A”  (Hibit  &  Marx,  1 994).  MEDA  supports  human-centered  error 
investigation  in  an  attempt  to  encourage  users  to  change  their  paradigm  about 
investigations  of  maintenance  error.  MEDA  is  based  on  a  maintenance  system  model 
where  contributing  factors  are  identified  at  each  of  four  encompassing  stages:  (1)  the 
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individual  mechanic,  (2)  the  mechanics  immediate  work  environment,  (3)  the  supervision 
provided  to  the  mechanic,  and  (4)  the  organizational  climate  set  for  the  mechanic 
(Boeing,  1997). 

Boeing  collected  in-flight  shutdown  (IFSD)  due  to  maintenance  error  data  from 
15  airlines  with  500,000  to  1  million  engine  hours  of  Boeing  aircraft  between  1983  and 
1993  (Boeing,  1997).  Bach  of  these  errors  was  assigned  a  causal  factor  before  being 
added  to  the  database.  Success  would  be  achieved  by  incorporating  this  system  into  an 
organization’s  everyday  operations  with  internal  management  of  maintenance  error 
providing  the  best  return  (Goglia,  J.,  National  Transportation  Safety  Board,  personal 
communication  with  Schmidt,  J.,  1998).  MEDA  is  based  on  process  improvement  and 
discourages  the  practice  of  simply  punishing  the  person  who  commits  the  error. 
Investigators  establish  contributing  factors  to  the  event  and  recommendations  for  process 
improvement,  all  of  which  are  added  to  the  MEDA  database.  Once  improvements  have 
been  made,  this  information  is  provided  to  all  affected  employees  (Boeing,  1 997). 

Success  has  been  cited  by  organizations  using  MEDA,  e.g.,  reduction  in 
maintenance  related  incidents,  improved  maintenance  practices  (Sargent  &  Smith,  1999). 
However,  Marx  (1998)  notes  that  MEDA  (and  human  factors  based  investigation 
methods  in  general)  is  not  being  widely  used.  Of  92  carriers  using  MEDA,  only  6  are  in 
the  United  States.  Placing  blame  on  workers,  not  transcending  proximate  causes, 
emphasizing  the  static  who,  what,  when  variables,  not  searching  for  underlying  causes, 
and  being  only  a  philosophy  vice  an  integrated  solution  were  all  cited  as  reasons  for  not 
using  MEDA  and  other  similar  tools  (Goglia,  J.,  National  Transportation  Safety  Board, 
personal  communication  with  Schmidt,  J.,  1998). 
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Both  Galaxy  and  BF  Goodrich  have  created  software  applications  for  MEDA  to 
transform  it  from  a  pencil  and  paper  collection  method  to  the  information  age.  Galaxy 
developed  “TEAM”--Tools  for  Error  Analysis  in  Maintenance 

(www.galaxyscientific.com,  2000)  and  BF  Goodrich  (BF  Goodrich,  1997)  followed  with 
a  hybrid  system  that  incorporates  MEDA  and  another  system  called  Aurora 
(www.hfskyway.gov,  1999).  These  applications  allow  the  user  to  collect,  organize, 
analyze,  and  report  data  through  an  interactive  graphical  user  interface  system.  Users  are 
able  to  enter  new  or  update  existing  error  data,  create  reports  (e.g.,  MEDA  forms, 
contributing  factors/error  summaries,  and  audit  information/checklists),  and  update 
information  on  corrective  actions  being  taken. 

(8)  Sharing  Maintenance  Error  Data 

The  Flight  Operational  Quality  Assurance  (FOQA)  program  is  a  voluntary,  but 
Federal  Aviation  Administration  (FAA)  sanctioned  effort,  to  share  aggregated  safety 
data,  continuous  from  flight  data  recorders,  across  commercial  airline  carriers 
(www.faa.gov,  2000).  The  intent  is  to  provide  a  means  to  examine  industry  wide  trends 
and  use  the  derived  information  to  enhance  training  of  personnel,  operational  procedures, 
maintenance  and  engineering,  air  traffic  control,  and  airport  surface  safety.  FAA 
Administrator  Jane  Garvey  stated,  “FOQA  programs  are  already  producing  the  hard  data 
we  need  to  identify  safety  records,  target  potential  problems,  and  make  corrections  before 
accidents  happen  (Reuters,  2000).”  Data  to  be  collected  includes  Ground  Proximity 
Warning  System  (GPWS)  warnings,  excessive  rotation  rates  on  take  off,  un-stabilized 
approaches,  hard  landings,  and  compliance  with  standard  operating  procedures. 
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Additional  information  includes  fuel  efficiency,  out-of  trim  airframe  configuration 
identification,  engine  condition,  compliance  with  noise  abatement,  rough  runway 
surfaces,  and  aircraft  structural  fatigue  (Garvey,  1998). 

Presently,  230  total  aircraft  consisting  of  13  aircraft  types  are  electronically 
collecting/sharing  FOQA  data  (Reuters,  2000).  An  impetus  for  sharing  under  the  banner 
of  safety  is  that  shared  FOQA  data  is  not  used  for  enforcement  purposes  except  under 
egregious  circumstances.  This  cooperation  has  not  been  as  successful  in  extending  to  the 
hangar  bay  and  flight  line  in  terms  of  maintenance  and  sharing  error  and  incident  data. 
The  FAA  and  NTSB  both  concur  that  this  is  an  essential  part  of  the  overall  safety 
equation  for  increasing  commercial  aviation  safety.  One  major  problem  standing  in  the 
way  is  having  a  common  process/taxonomy  for  capturing,  recording,  and  archiving 
accident/incident/error  data  for  aggregate  and  trend  analysis  (Goglia,  J.,  National 
Transportation  Safety  Board,  personal  communication  with  Schmidt,  J.,  1998). 

Boeing’s  MED  A,  Galaxy’s  TEAM  (Tools  for  Error  Analysis  in  Maintenance)  tool 
and  BF  Goodrich’s  MEDA  software  tool  all  attempt  to  achieve  a  vehicle  for  not  only 
capturing  mishap  information,  but  also  to  share  data  across  the  industry  (Goglia,  J., 
National  Transportation  Safety  Board,  personal  communication  with  Schmidt,  J.,  1998). 
Unfortunately,  though  used  by  some  of  Boeing’s  customers  (e.g.,  BF  Goodrich  in  its  re¬ 
work  facility),  MEDA  has  not  been  adopted  as  an  industry  standard  (Marx,  1998).  This 
is  due,  in  part,  to  the  in  house  requirement  to  staff  such  an  initiative,  the  training 
requirements  involved,  and  issues  related  to  unions,  culpability,  etc.  The  latter  is  tied  to 
the  emphasis  on  the  immediate  act  of  the  person  and  not  the  organizational  and  work 
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settings  that  have  contributed  to  it.  Consequently,  a  need  exists  to  develop  a  tool 
encompassing  accident  data  collection,  organization,  analysis,  and  reporting. 

D.  TOOL  DESIGN  CONSIDERATIONS 

(1)  System  Design 

The  usability  of  a  product  is  directly  linked  to  the  user  interface.  A  user  interface 
that  is  easy  to  learn  and  use  will  have  favorable  usability  evaluations.  To  maximize  the 
usability  of  an  interface,  Shneiderman  (1997)  proposed  eight  golden  rules  of  graphical 
user  interface  design:  (1)  consistency,  (2)  shortcuts  for  frequent  users,  (3)  informative 
feedback,  (4)  dialogs  designed  to  yield  closure,  (5)  error  prevention  and  simple  error 
handling,  (6)  simple  reversal  of  actions,  (7)  internal  locus  of  control,  and  (8)  reduced 
short-term  memory  load.  A  sense  of  comprehension  and  competence  among  the  users 
will  be  the  end  result  of  following  these  rules.  If  users  feel  familiar  and  competent  with 
systems,  they  will  more  likely  them  highly.  (Shneiderman,  1997). 

Consistency  can  relate  to  many  aspects  of  the  system:  terminology,  color,  layout, 
input,  display  formats,  etc.  Though  consistency  cannot  always  be  maintained  across  all 
dimensions  of  a  system,  symbology  and  methods  of  interaction  should  be  consistent. 
Frequent  users  can  reduce  the  number  interactions  required  for  a  specific  result  through 
shortcuts.  These  will  also  increase  the  pace  of  interaction.  Information  feedback  can 
vary  in  degree  depending  on  the  frequency  and  severity  of  action  involved  and  allow 
users  to  more  fully  understand  their  current  status  by  immersing  them  in  the  graphical 
environment.  Designing  dialogs  to  yield  closure  can  be  achieved  by  grouping  actions  to 
set  up  a  natural  flow  through  the  user’s  tasks.  This  gives  the  user  a  better  sense  of  the 
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actions  as  they  are  being  performed  and  a  better  awareness  of  a  sequence’s  closure. 
(Shneiderman,  1997). 

Error  prevention/handling  allows  the  user  to  maintain  confidence  in  the  system’s 
fidelity  and  ability  to  recover  from  minor  fluctuations.  Reversing  an  action  allows  users 
to  recover  from  their  mistakes  easily,  reducing  stress  or  anxiety  of  operating  within  the 
system.  Designing  an  internal  locus  of  control  allows  the  users  to  be  in  command  of  the 
environment  and  not  vice  versa.  Users  would  not  experience  autonomous  movement 
within  the  environment  or  a  drastic  change  of  the  visual  orientation.  Finally,  the 
reduction  of  short-term  memory  load  includes  access  to  integrated  assistance  information 
(e.g.,  cues,  mnemonics,  standardized  sequence  of  actions).  (Shneiderman,  1997). 

(2)  Usability  Study 

Usability  testing  is  a  systematic  means  of  observing  the  ease  of  use  of  a  product 
and  collecting  related  data.  Dumas  &  Redish  (1994)  identified  three  tenets  of  usability 
studies:  (1)  It  should  be  used  to  diagnose  problems  vice  determining  that  the  product  is 
flawless;  (2)  Usability  testing  should  be  employed  early  and  often  during  the 
development  cycle,  and  (3)  It  is  part  of  a  process  that  focuses  on  usability  throughout 
design  and  development. 

A  thorough  testing  plan  needs  to  be  developed  in  order  to  best  incorporate 
usability  into  the  development  process.  Several  factors  must  be  addressed  in  an 
evaluation  plan  (Shneiderman,  1997;  Nielsen,  1993;  Hix  &  Hartson,  1993;  Preece,  et.  al., 
1994;  Newman  &  Lamming,  1995).  First,  the  current  stage  of  the  design  will  determine 
the  requirements  for  testing,  with  different  conditions  for  different  stages.  Second,  the 
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criticality  of  the  environment  helps  to  decide  the  objectives  of  the  test.  Finally,  other 
factors,  such  as  the  novelty  of  the  project,  number  of  expected  users,  time  available,  cost 
of  the  project,  available  resources  (e.g.,  time,  money,  people),  and  experience  of  the 
testers  all  play  a  role  in  defining  the  usability  study. 

A  usability  study  not  only  maximizes  the  usability  of  the  system,  but  ensures 
contractual  requirements  have  been  completed  and  to  provide  evidence  of  testing  in  cases 
of  lawsuits  or  if  legal  issues  arise.  Errors  in  a  system  will  be  tolerated  to  varying  degrees 
dependent  upon  the  need  to  bring  the  system  to  full  operational  use  and  the  impact  the 
errors  may  have  during  that  time.  However,  a  system  is  more  difficult  to  test  as 
increasing  amounts  of  input  are  required,  yet  these  tests  are  increasingly  needed. 
(Shneiderman,  1997). 

Some  (Nielsen  &  Mack,  1994)  argue  for  an  expert  evaluation  of  a  system  to 
increase  a  product’s  usability.  Formal  reviews  can  provide  more  useful  information 
when  compared  to  informal  demonstrations  of  a  product.  Thus,  system  design  and 
testing  should  employ  expert  reviewers  who  typically  produce  a  report  detailing  problems 
and  recommendations  for  improvement.  These  reviews  may  take  the  form  of  heuristic 
evaluation,  guideline  review,  consistency  inspection,  cognitive-walkthrough,  and  formal 
usability  inspection  (see  Table  3).  Even  if  the  experts  are  reviewing  unfamiliar  systems 
and  technologies,  they  still  provide  a  fresh  look  at  a  system  and  are  useful  in  evaluating 
system  development. 
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Table  3:  Expert  Review  Methods  (Shneiderman,  1997) 


Expert-Review 

Method 

Description 

Heuristic  evaluation 

Expert  reviewers  critique  an  interface  to  determine  conformance 
with  a  short  list  of  design  heuristics  such  as  the  eight  golden  rules. 

Guideline  review 

The  interface  is  checked  for  conformance  with  the  organizational 
or  other  guidelines  document. 

Consistency 

inspection 

Experts  verify  consistency  across  a  family  of  interfaces,  checking 
for  consistency  of  terminology,  color,  layout,  input  and  output 
formats,  within  the  interfaces  as  well  as  in  the  training  materials 
and  online  help. 

Cognitive- 

walkthrough 

Experts  simulate  users  walking  through  the  interface  to  carry  out 
typical  tasks.  Simulating  the  day  in  the  life  of  the  user  should  be 
part  of  the  evaluation. 

Formal  usability 
inspection 

HI _ *  1 _ _  +  r\r\rr 

Experts  hold  courtroom-style  meeting,  with  a  moderator  to  judge, 
to  present  the  interface  and  to  discuss  its  merits  and  weaknesses. 

Shneiderman,  1997 


Usability  studies  take  additional  forms,  such  as  discount  usability  engineering; 
“short  and  sweet”  approaches  to  task  analysis,  prototyping,  and  testing.  Another  style  of 
study  is  a  field  study  conducted  in  actual  work  environments  in  order  to  achieve  realistic, 
user  evaluation.  Alternatively,  beta  testing  challenges  actual  users  break  the  system. 

This  method  can  expedite  the  development  process  and  correct  errors  missed  through 
conventional  testing.  Usability  testing  can  lack  comprehensive  system  evaluation  due  to 
time  constraints  and  also  tends  to  emphasize  first-time  usage  (Shneiderman,  1997).  Thus 
usability  studies  must  be  supplemented  with  other  methods  of  evaluation,  such  as  expert 
review.  (Shneiderman,  1997). 

An  important  decision  to  be  made  when  planning  a  usability  study  is  how  long  the 
test  should  take  (Dumas  &  Redish,  1994).  If  the  study  is  conducted  as  an  integrated  part 
of  the  design  process  and  is  not  simply  being  conducted  on  a  completed  system,  then  the 
test  length  should  be  reduced  to  a  level  to  both  obtain  necessary  information  and  not  be  a 

burden.  Dumas  &  Redish  (1994)  place  traditional  testing  into  four  categories. 
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(1)  Formal  testing  with  comprehensive  test  reports  requires  eight  to  twelve  weeks.  (2)  If 
a  strong  collaboration  is  exhibited  among  team  members  and  a  shortened  report  format  is 
used,  then  four  to  six  weeks  are  required.  (3)  If  a  particular  part  of  the  system  is  to  be 
studied  with  well  established  procedures,  then  one  week  may  be  appropriate.  (4)  Finally, 
just-in-time  testing  can  provide  useful  information  in  a  few  days,  if  necessary,  but  is 
discouraged. 

Dumas  &  Redish  (1994)  contend  proper  planning  includes  defining  goals, 
identifying  concerns,  recruiting  individuals  and  their  participation,  creating  and 
organizing  tasks/task  scenarios,  deciding  on  usability  measures,  preparing  test  materials 
and  test  environment,  and  conducting  a  pilot  test.  Defining  goals  and  identifying 
concerns  is  a  three-stage  process:  (1)  making  choices  among  goals  and  concerns; 

(2)  moving  from  general  to  specific  concerns  helping  to  mold  concerns  into  quantitative 
objectives;  and  (3)  understanding  the  source  of  the  goals  and  concerns  allowing  the 
usability  engineer  to  better  develop  the  testing  scenarios  and  tasks.  Developed  user 
profiles,  preferably  prior  to  system  design  and  usability  testing,  can  provide  the  basis  for 
deciding  upon  participants  in  a  study.  The  realities  of  time  and  budget  constraints  often 
result  in  usability  studies  having  less  participants  than  usability  engineers  (10-12)  or 
statisticians  (at  least  36-48)  may  desire.  (Dumas  &  Redish,  1994). 

(3)  Human-Computer  Interface  (HCI)  Design  Issues 

Brown  (1989)  posits  that  information  database  management  system  should  be 
considered  a  simple  tool,  simplifying  rather  than  complicating  the  tasks  of  the  user.  Thus 
the  design  of  the  system  should  reduce  mental  processing  operations  (learning  complex 
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commands/syntax,  memorizing  codes/abbreviations,  translating  data  into  other 
units/formats)  required  to  operate  the  tool.  He  feels  the  allocation  of  functions,  one  of  the 
most  important  categories  of  design  decisions,  should  be  performed  based  on  the 
capabilities  of  both  the  user  and  the  system.  Brown  (1989)  suggests  five  “user”  rules  for 
allocating  functions  in  routine  interface  design:  (1)  Minimize  the  amount  of  procedural 
memorization.  (2)  Reduce  mental  manipulation.  Data  should  be  presented  clearly  and  in 
a  useable  form.  (3)  Minimize  manual  entries.  Allow  the  user  to  select  from  a  displayed 
list  vice  forcing  manually  entries.  (4)  Offer  computer  aids  (e.g.,  checklists,  summaiy 
displays,  and  help  functions)  to  reduce  both  required  mental  processing  and  the  need  for 
executing  complex,  multi-step  procedures.  (5)  Use  computer  algorithms  to  process  and 
present  complex  data. 

Mental  models,  or  a  cognitive  representation  of  the  internal  parts  and  operations 
of  a  system,  are  another  important  part  of  successful  HCI.  The  user’s  mental  model 
allows  him/her  to  predict  the  appropriate  procedure  for  a  desired  outcome,  even  if  the 
procedure  has  been  forgotten.  Thus,  a  user’s  mental  model  can  give  the  user  an 
understanding  of  how  a  system  works  and  develop  and  refine  his/her  knowledge  when 
learning  about  or  using  the  system.  Several  principles  for  mental  models  should  be 
incorporated  into  a  system:  consistency  (both  actions  and  classes  of  actions),  physical 
analogies,  user  expectations,  and  stimulus-response  compatibility.  (Brown,  1989). 

The  designer  should  provide  for  a  balance  of  ease  of  learning,  ease  of  use,  and 
functionality  in  a  system.  Brown  (1989)  identifies  four  techniques  to  ensure  this  balance 
is  maintained:  (1)  incorporate  needs  of  novices,  experts,  and  intermittent  users  into  the 
design,  (2)  avoid  excess  functionality,  (3)  provide  multiple  paths,  and  (4)  progressive 
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disclosure  and  graceful  evolution  design.  These  procedures  will  ensure  equilibrium  is 
established  in  a  system. 


E.  SUMMARY 

A  well-defined,  systematic  accident  investigation,  analysis,  and  reporting  process 
is  critical  in  the  effort  to  reduce  the  Naval  Aviation  mishap  rate.  Yet,  no  one  good 
universal  system  currently  exists  (Marx,  1998).  Such  a  tool  must  have  a  reporting  system 
relying  on  a  solid  data  collection  process  with  the  ability  to  readily  access  the  stored  data. 
Many  times,  mishap  data  is  lost  or  not  used,  often  leading  to  yet  another  incident.  The 
goal  for  such  a  tool  is  to  use  the  data  for  prevention  of  future  accidents. 

Effectively  addressing  human  error  issues  can  greatly  increase  safety  levels. 
Several  robust  approaches  to  examining  mishaps  involving  human  error  achieve  this 
goal:  Heinrich’s  “Domino”  Theory,  Edwards’  “SHEL”  Model,  and  Reason’s  “Swiss 
Cheese  Model.  Once  an  approach  is  examined,  a  means  of  bridging  the  gap  between 
theory  and  user  must  be  made.  The  Navy’s  Human  Factors  Analysis  &  Classification 
System  (HFACS)  and  its  Maintenance  Extension  offshoot  (HFACS-ME)  appear  to  be 
potential  approaches  for  investigating,  reporting  and  analyzing  maintenance  error. 

However,  the  designer  must  maximize  the  usability  of  the  interface.  This  can  be 
accomplished  by  following  Shneiderman’s  (1997)  eight  golden  rules  of  graphical  user 
interface  design.  Once  the  system  is  designed,  a  usability  study  with  a  prototype  tool 
ensures  the  product  is  ready  for  general  use  by  testing  it  with  a  selected  group  of  users. 
Finally,  Human-Computer  Interface  issues  must  be  addressed  to  simplify  rather  than 
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complicate  the  life  of  the  user.  Once  the  above  goals  are  met,  the  system  is  ready  to  meet 
its  challenge. 


III.  METHODS 


A.  RESEARCH  APPROACH 

A  software  desktop  analysis  and  reporting  tool  for  maintenance  error  in  aviation 
would  greatly  facilitate  Naval  Aviation’s  effort  to  capture  human  factors  in  mishaps  and 
develop  interventions.  The  Maintenance  Error  Information  Management  System 
(MEIMS)  is  a  computer-based  prototype  tool  designed  using  Microsoft  Access  97  and 
Visual  Basic  6.0.  The  prototype  utilizes  a  database  derived  from  the  Naval  Safety 
Center’s  Safety  Information  Management  System  (SIMS)  database,  which  contains 
information  on  over  600  maintenance  error  related  mishaps  that  occurred  between  1989 
and  1 999.  The  system  has  a  graphical  user  interface  (GUI)  that  allows  the  end-user  to 
operate  the  system  with  basic  computer  skills. 

The  prototype  was  distributed  to  a  representative  sample  of  potential  end-users. 
The  participants  were  provided  a  prepared  task  list  that  required  them  to  navigate  through 
and  utilize  features  of  the  tool.  At  the  completion  of  the  task  list,  the  participants  had 
viewed  and  used  all  portions  of  the  prototype  tool,  and  completed  an  exit  survey 
composed  of  questions  pertaining  to  demographic  background  information  and  both 
objective  and  open-ended  items  to  elicit  the  participants’  views  of  the  usability  of  the 
system  and  value  of  both  the  system  and  the  data.  The  objective  data  was  transcribed  into 
a  Microsoft  Excel  spreadsheet  for  analysis  while  a  content  analysis  was  conducted  on  the 
open-ended  survey  questions.  Note,  the  exit  survey  used  only  five  Likert  style  questions 
because  the  major  focus  of  the  effort  was  the  creation  of  the  prototype  tool  vice  the 
usability  study.  The  questions  were  shaped  intuitively  and  are  considered  to  be  simply 
the  first  stage  of  developing  a  formalized  post-prototype  tool. 
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B.  DESCRIPTION  OF  MAINTENANCE  EXTENSION  INFORMATION 
MANAGEMENT  SYSTEM  (MEIMS) 

(1)  Overview 

The  Maintenance  Extension  Information  System  (MEIMS)  prototype  tool  was 
designed  to  allow  the  user  to  have  access  to  the  data  base  via  three  functional  tools:  (a) 
sorting  the  data  by  queries,  (b)  obtaining  output  from  the  data  base  via  written  reports 
and  graphical  displays,  and  (c)  providing  input  to  the  data  base  through  the  addition  of 
new  data.  Each  function  was  displayed  on  separate  pages  with  interactive  controls 
providing  user-prototype  interaction.  The  following  paragraphs  provide  a  description  of 
the  prototype.  A  completion  description  of  all  of  the  displays  is  found  in  Appendix  A. 

(2)  Main  Menu 

The  Main  Menu  of  the  prototype  allows  the  user  to  select  (left  click)  one  of  five 
different  options:  (a)  Query  Menu,  (b)  Report  Menu,  (c)  Expert  Graph  Menu,  (d)  Adding 
New  Data,  and  (e)  Exiting  the  system  (see  Figure  Al).  Help  is  provided  to  the  user  on 
this  and  all  pages  in  the  form  of  “control  tips”  (i.e.,  brief  descriptions)  when  the  mouse 
arrow  is  placed  over  a  control  (i.e.,  text  box,  command  button,  etc.)  (see  Figure  A5). 
Additional  help  is  found  in  the  “status  bar”  at  the  bottom  of  the  screen  when  a  control  is 
highlighted. 

(3)  Query  Menu 

The  Query  Menu  provides  the  user  two  manners  of  output  (see  Figure  A2).  The 
first  is  through  the  selection  of  one  of  eight  command  buttons  to  sort  the  data  base  by  one 
or  more  of  its  fields:  aircraft  model  (F-14,  H-46,  etc.),  aircraft  type  (tactical  aircraft 
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(TACAIR),  helicopters,  heavy  aircraft,  trainers,  and  others),  branch  of  service  of  the 
aircraft  (USN,  USMC),  location  of  the  mishap  (ashore,  embarked,  and  detached),  mishap 
classification  (A,  B,  or  C),  mishap  type  (Flight  Mishap  (FM),  Flight-Related  Mishap 
(FRM),  or  Aircraft-Ground  Mishap  (AGM)),  calendar  year  of  the  mishap  (1989-1999), 
and  any  combination  of  the  above  (Multiple  Criteria  selection). 

When  a  single  category  control  is  selected  a  sub-menu  appears  where  the  user  can 
define  the  exact  description  of  the  category  via  a  combo  box  (see  Figures  A3  and  A6). 
Upon  selecting  the  View  Selection”  control  a  Maintenance  Mishap  Query  window  is 
displayed  revealing  each  instance  (mishap)  of  the  selected  description  (see  Figure  A4). 

In  addition,  the  user  may  page  through  all  mishaps  of  the  selection  by  selecting  the  right 
arrow  on  the  bottom  of  the  window  (see  Figures  A4  and  A5).  The  data  for  each  mishap 
is  displayed  in  text  boxes,  with  the  selected  category  denoted  with  blue  background  (see 
Figure  A4).  Additionally,  maintenance  related  contributing  factors  to  the  mishap  with 
their  HFACS-ME  codes  are  displayed  at  the  bottom  of  the  window.  Selecting  the 
“Multiple  Criteria”  control  on  the  Query  Menu  will  allow  the  user  to  further  define  the 
data  base  by  selecting  any  or  all  of  the  seven  solo  categories  (see  Figures  A7  and  A8).  A 
Multiple  Criteria  sub-menu  appears  and  allows  the  user  to  “check”  the  desired  categories 
and  further  define  them  by  selecting  criteria  provided  in  combo  boxes  on  the  sub-menu. 

In  these  cases,  all  of  the  selected  categories  will  have  a  blue  background  on  the  resulting 
Maintenance  Mishap  Query  window  (see  Figure  A9). 

Throughout  the  prototype,  “Define  HFACS  Codes”  controls  are  displayed.  When 
selected  they  will  provide  a  summary  sheet  of  the  level  one,  two,  and  three  HFACS-ME 
codes,  with  each  acronym  defined  (see  Figures  A9  and  A10).  In  addition,  at  any  time  the 
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user  may  close  a  window  and  return  to  the  previous  sub-menu  by  selecting  the  “Close 
Form”  or  “<Back”  control  (see  Figure  A5).  Each  of  the  four  primary  menus  has  a 
“Return  to  Main  Menu”  control  which  returns  the  user  to  the  Main  Menu  when  selected. 

The  second  output  provided  by  the  Query  Menu  is  the  HFACS-ME  Summary 
Form  (see  Figure  A1 1).  This  form  allows  the  user  to  view  a  summary  of  data  categorized 
by  HFACS-ME  levels  one,  two,  and  three.  The  user  may  also  further  define  the  output 
by  selecting  any  or  all  of  the  previously  mentioned  seven  categories  via  combo  boxes 
(see  Figure  A12). 

(4)  Report  Menu 

Report  Menu  is  the  second  option  a  user  may  select  from  the  Main  Menu  (see 
Figure  A 13).  The  Report  Menu  offers  eight  reports  which  provide  data  listing  the  total 
number  of  mishaps  and  the  number  and  percentages  of  mishaps  by  HFACS-ME  levels 
one,  two,  and  three  (see  Figure  A15).  The  user  may  select  from  the  following 
distribution  presentations:  all  mishaps,  aircraft  model,  mishap  class,  mishap  type,  mishap 
class  by  mishap  type,  branch  of  service,  mishap  location,  and  chronological  listing  by 
aircraft  model  (see  Figures  A14  -  A16).  All  reports  are  closed  by  selecting  the  “Close” 
control  at  the  top  of  the  window. 

(5)  Expert  Graph  Menu 

The  third  option  from  the  Main  Menu  is  the  Expert  Graph  Menu  (see  Figure 
A1 7).  The  user  may  create  a  two-axis,  three-dimensional  graph  presentation.  The  x-  and 
y-axes  are  populated  with  one  of  the  following  categories:  aircraft  model,  aircraft  type. 
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mishap  location,  branch  of  service,  mishap  class,  mishap  type,  HFACS-ME  level  one, 
HFACS-ME  level  two,  and  HFACS-ME  level  three  (see  Figure  A18).  The  user  may  then 
select  one  or  more  of  the  fields  from  each  axis  category  sub-menu  via  a  combo  box  (see 
Figures  A18  and  A19).  The  resultant  graph  is  presented  in  a  three-dimensional,  multi¬ 
colored  view  (see  Figure  A20). 

(6)  Add  New  Data  Menu 

The  user  may  populate  the  data  base  by  selecting  the  “Add  New  Data”  control  on 
the  Main  Menu  (see  Figure  A21).  The  user  must  then  fill  in  an  “Enter  New  Maintenance 
Mishap  Data”  form  with  the  following  fields:  mishap  class,  mishap  type,  date  of  mishap, 
aircraft  type/model  (F-14,  H-46,  etc.),  aircraft  category  (TACAIR,  helo,  etc.),  branch  of 
service,  location  of  mishap,  and  a  brief  description  of  the  mishap.  The  prototype 
automatically  assigns  a  Mishap  Identification  Number.  A  sub-menu  on  the  form  asks  the 
user  to  input  mishap  factor  ’  data.  This  information  includes:  a  brief  description  of  the 
factors  and  the  HFACS-ME  level  three  code.  Upon  selection  of  the  level  three  code,  the 
levels  one  and  two  codes  and  descriptions  and  level  three  descriptions  are  automatically 
entered  by  the  prototype,  as  is  the  Factor  Identification  Number  (see  Figure  A23).  The 
user  can  enter  an  additional  factor  by  selecting  the  “Add  New  Factor”  control  (see  Figure 
A23).  After  all  mishap  data  is  entered,  the  “Close  Form”  control  is  selected  (see  Figure 
A23).  The  “Final  Data  Entry”  form  appears  and  asks  the  user  to  select  “Enter”  and  wait 
for  the  “Done”  box  to  be  checked  to  show  successful  data  entry. 
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C.  DATA  COLLECTION 


(1)  Subjects/Participants 

Students  (n=42)  attending  the  Aviation  Safety  Officer  (ASO)  course  at  the  School 
Aviation  Safety,  at  the  Naval  Postgraduate  School  in  Monterey,  CA  participated  in  the 
study.  Individual  activities  are  allocated  class  spaces  to  determine  enrollment  in  the  ASO 
course  and,  consequently,  attendee  demographics  represent  a  wide  cross  section  of  Naval 
Aviators  and  Naval  Flight  Officers,  Coast  Guard  and  other  DOD  officers,  Flight 
Surgeons  and  Aeromedical  Safety  Officers,  and  foreign  nationals  from  all  aircraft 
communities.  ASO  course  graduates  are  responsible  for  the  management  and 
implementation  of  squadron  safety  programs  to  include  mishaps  and  include 
investigations  and  reporting.  They  are  likely  to  be  one  of  the  primary  end-users  of  the 
tool.  Participant  demographics  were  characterized  by  aviation  background,  computer 
experience,  and  availability  of  software  and  hardware  systems  used  in  the  Navy  and 
Marine  Corps. 

(2)  Apparatus 

The  ASO  students  had  access  to  three  computer  labs  (Pentium  level)  at  the  School 
of  Aviation  Safety  via  login  identification  and  password  to  a  group  account.  The 
computer  in  each  lab  had  a  full  prototype  functioning  desktop  analysis  and  reporting  tool 
loaded  onto  it.  After  a  participant  gained  access  to  the  computer,  the  “MEIMS”  icon  was 
found  on  the  computer  desktop  and  selected  to  open  the  application. 

The  prototype  was  developed  using  Microsoft  Access  2000  and  Visual  Basic  6.0 
and  consisted  of  four  sections:  database  queries,  reports,  graphic  presentations,  and  data 
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entry.  Each  section  was  divided  into  further  subsections  allowing  the  participant  to  more 
specifically  design  his  access  to  the  database.  This  allowed  the  participant  to  achieve  the 
four  functional  requirements  for  the  software  tool:  data  collection,  organization,  analysis, 
and  reporting  (see  Appendix  A  for  a  more  complete  description  of  the  prototype). 

(3)  Instrument 

A  participant  usability  survey  was  constructed  by  the  author  consisting  of  three 
parts:  (1)  Participant  demographics,  (2)  Likert  type  assessment  questions,  and  (3)  Open- 
ended  items.  Collection  of  demographic  information  was  accomplished  through  the 
participant  selecting  from  a  list  of  descriptors  (rank,  branch  of  service,  experience 
level/years  of  service).  Survey  questions  were  designed  to  determine  if  the  prototype 
software  tool  met  participant  investigation,  reporting,  and  analysis  requirements.  The 
Likert  questions  used  a  five  point  rating  scale  with  verbal  anchors:  Strongly  Agree, 

Agree,  Neutral,  Disagree,  and  Strongly  Disagree.  Open-ended  questions  were  included 
to  gain  subjective  responses  on  the  overall  impression  of  the  prototype  software  tool, 
recommendations  for  improvement,  and  comments  on  areas  not  adequately  covered  by 
parts  one  and  two  of  the  survey  (see  Appendix  B). 

(4)  Procedure 

The  prototype  software  desktop  analysis  and  reporting  tool  containing  a  database 
derived  from  Naval  Safety  Center  maintenance  mishaps  was  loaded  on  seven  computers 
in  three  computer  labs  with  24-hour  accessibility.  A  MEIMS  icon  was  placed  on  each 
computer  desktop  page  to  allow  access  into  the  program. 
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Prototype  testing  occurred  over  a  span  of  one  week.  Participants  were  given  a 
group  orientation  briefing  by  the  author  on  the  purpose,  goals,  and  procedures  of  the 
prototype  including  a  projected  computer  demonstration.  In  addition,  they  were  given  a 
10-page  guide  to  walk  them  through  prototype  testing  (see  Appendices  B  and  C).  The 
guide  consisted  of: 

•  Instructions  for  Accessing  the  Prototype  Software  Tool  -information  to  log 
on  and  open  the  prototype  (see  Appendix  B). 

•  Prototype  Software  Tool  Task  List— a  series  of  planned  navigation  routes 
within  the  prototype  whereby  the  participant  would  be  able  to  view  the  entire 
system  (see  Appendix  B). 

•  Participant’s  Impression  of  the  Prototype  Software  Tool/Exit  Survey 
(see  Appendix  C). 

The  author,  with  full  knowledge  of  both  prototype  MEIMS  tool  and  Microsoft 
Access  procedures  took  six  minutes  to  complete  the  testing.  It  was  expected  that  each 
participant  would  need  15-20  minutes  to  complete  the  process.  Though  information  on 
time  to  navigate  for  each  individual  was  not  taken,  informal  feedback  to  the  author 
indicated  a  range  of  15-30  minutes  with  the  longer  times  being  needed  for  those  with  less 
computer  and  Microsoft  Access  experience.  One  computer  was  a  lower-end  model  (i.e., 
Pentium  I,  133  MHz,  4mb  RAM)  which  caused  some  functions  not  to  work  properly, 
most  notably  the  expert  graphing  option.  At  the  completion  of  the  task  list,  participants 
viewed  all  portions  of  the  prototype  system,  and  formed  an  opinion  on  its  effectiveness. 
Participants  then  completed  an  exit  survey  composed  of  demographic  background 
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questions  and  perusal  of  the  prototype  system.  Surveys  were  all  submitted  through  a 
drop  box  provided  in  a  common  area. 


D.  DATA  TABULATION 

The  data  was  transcribed  from  the  survey  onto  a  Microsoft  Excel  2000 
spreadsheet.  The  Likert  questions,  based  on  a  five-point  scale,  were  coded  into  Excel, 
using  numbers  1  through  5  corresponding  to  the  respective  anchors  (Strongly  Disagree, 
Disagree,  Neutral,  Agree,  and  Strongly  Agree).  Descriptive  statistics  were  generated 
using  Excel  functions  including  the  mean,  standard  deviation,  range,  and  frequency 
distribution  of  the  collected  data.  Content  analysis  was  conducted  on  the  responses 
provided  from  the  open-ended  survey  questions.  The  categorization  of  participants  by 
participant  aircraft  maintenance  organization  type  and  computer/software  application 
experience  level  were  noted.  However,  due  to  no  noticeable  differences  between 
categories,  all  subsequent  analysis  was  performed  on  all  participants  as  a  single  group. 


E.  DATA  ANALYSIS 

Basic  and  general  information  about  the  demographic  and  question  results  were 
depicted  using  descriptive  non-parametric  analysis  is  conducted  on  the  survey  data  in 
order  to.  Basic  summary  statistics  are  developed  with  results  including  demographic 
information  and  satisfaction  levels  with  the  prototype.  Analysis  of  the  data  is  performed 
using  the  functions  of  Microsoft  Excel. 
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IV.  RESULTS 


A.  SAMPLE 

The  13  item  exit  survey  was  administered  to  45  participants  from  a  School  of 
Aviation  Safety  “Aviation  Safety  Officer”  course  with  a  response  rate  of  95.6%  (n=43). 
Each  Naval  Aviation  command  is  required  to  have  an  officer  trained  by  the  Safety 
School.  Thus  the  participants  were  designated  Naval  Aviators  and  Naval  Flight  Officers 
and  represented  a  cross  section  of  the  aviation  commands  that  make  up  the  squadrons  in 
the  Navy  and  Marine  Corps. 

B.  DEMOGRAPHIC  INFORMATION 

The  material  collected  in  Part  I  of  the  exit  survey  consisted  of  demographic 
information  and  established  the  aviation  and  computer  experience  levels  of  each 
participant  had  both  with  computers  and  in  aviation.  The  information  is  later  used  to 
determine  if  experience  level  in  either  category  affected  a  participant’s  level  of 
satisfaction  and/or  impacted  the  usability  of  the  prototype  MEIMS  tool.  The  following 
paragraphs  characterize  the  survey  results  for  part  I. 

Question  one  revealed  that  39  of  the  participants  were  members  of  aviations  units 
that  performed  maintenance  at  the  squadron  level  (n  =  39,  90.7%).  The  remaining  four 
participants  were  either  members  of  higher-headquarter  staffs  (n  =  2;  4.7%)  or  units  that 
used  civilian  contract  personnel  to  perform  the  maintenance  (n  =  2;  4.7%).  Question  two 
indicated  that  all  but  one  participant  (n  =  42, 97.7%)  stated  they  had  at  least  two  years  of 
experience  using  a  computer.  The  remaining  participant  had  less  than  one  year  of 
computer  experience.  Question  three  determined  that  all  participants  (n  =  43;  100%) 
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were  users  of  Microsoft  Office,  while  minimal  numbers  had  used  either  Lotus  Notes  (n  = 
4;  9.3%)  or  Corel/Word  Perfect  (n  =  3;  7.0%).  Question  four  established  a  participant’s 
familiarity  with  different  software  applications,  greater  than  80  percent  stated  they  were 
familiar  with  at  least  one  of  the  following:  word  processing,  spreadsheet,  presentation, 
and  e-mail  (n  >=  35;  81.4%).  The  average  participant  was  familiar  with  approximately 
four  (4.3)  of  the  categories  (see  Table  4). 

Table  4.  Number  of  Participants  Familiar  with  Specific  Software  Applications 


(n=43) 


Word 

Processing 

Spread 

Sheet 

Presentations 

Graphic 

Software 

E-mail 

Database 

#  Familiar 

42 

37 

35 

11 

19 

% 

97.7 

86 

81.4 

25.6 

93 

44.2 

Question  five  revealed  the  normal  operating  system  (OS)  for  participants,  42 
(n  =  42;  97.7%)  responded  with  either  Windows  97/2000,  Windows  NT,  or  both  (see 
Table  5).  Two  participants  did  not  answer  the  question.  The  prototype  was  loaded  on 
computers  running  the  Windows  NT  operating  systems.  Participants  could  indicate  more 
than  one  OS. 


Table  5.  Normal  Operating  System  of  the  Participant  (n=41) 


Windows 

Windows  NT 

Mac 

Unix 

Linux 

Normal  OS,  # 

35 

29 

3 

4 

2 

% 

85.4 

70.7 

7.3 

9.9 

4.9 
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C.  PARTICIPANT  SATISFACTION  WITH  THE  PROTOTYPE  MEIMS  TOOL 
(1)  Responses  to  Impressions  and  Usability  Questions 

Part  II  of  the  exit  survey  examined  a  participant’s  impressions  of  the  usability  of 
the  prototype  MEIMS  tool  and  its  value  to  Naval  Aviation.  Participants  responded  to 
five  statements  using  Likert  type  responses  selecting  from  one  of  five  responses:  strongly 
agree,  agree,  neutral,  disagree,  and  strongly  disagree.  Values  of  five  through  one 
respectively  were  assigned  to  the  statements.  One  participant  did  not  respond  to  any  of 
the  statements.  The  participants  were  also  given  the  chance  to  make  subjective 
comments  on  any  of  the  five  statements. 

(a)  Statement  one  asked  whether  or  not  a  participant  found  the  prototype  to 
be  presented  in  a  logical  form.  The  histogram  of  the  frequency  distribution  for  statement 
one  is  presented  in  Figure  9.  The  mean  was  4.26,  standard  deviation  =  0.665,  range  =  4. 
Most  participants  (n  =  39;  92.7%)  agreed  that  the  prototype  was  designed  and  presented 
in  a  logical  fashion.  One  participant  did  not  select  a  response. 


"Is  in  a  logical  format" 


Agree  Disagree 


Figure  9:  Exit  Survey,  Part  II,  Statement  One,  Response  Distribution 
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(b)  Statement  two  asked  about  the  ease  of  navigation  of  the  prototype.  The 
histogram  of  the  frequency  distribution  for  statement  two  is  presented  in  Figure  10.  The 
mean  was  3.95,  standard  deviation  =  0.947,  range  =  5.  A  large  majority  of  the 
participants  (n  =  33;  80.5%)  agreed  that  the  prototype  was  easy  to  navigate.  Two 
participants  did  not  make  a  response  to  this  statement. 


"Is  easy  to  navigate" 


Strongly  Agree  Neutral  Disagree  Strongly 

A9ree  _  Disagree 


Figure  10:  Exit  Survey,  Part  II,  Statement  Two,  Response  Distribution 

(c)  Statement  three.  The  participants  were  asked  whether  they  felt 
MEIMS  was  “interesting.”  The  histogram  of  the  frequency  distribution  for  statement 
three  is  presented  in  Figure  1 1 .  The  mean  was  4.07,  standard  deviation  =  0.8 1 8,  range  = 
4.  A  large  majority  of  the  participants  (n  =  33;  80.5%)  indicated  the  prototype  was  of 
interest  to  them.  Two  participants  did  not  select  a  response. 


56 


Figure  11:  Exit  Survey,  Part  II,  Statement  Three,  Response  Distribution 


(d)  Statement  four  asked  about  the  relevance  of  the  prototype  to  aviation 
maintenance  operations.  The  histogram  of  the  frequency  distribution  for  statement  four 
is  presented  in  Figure  12.  The  mean  was  4.40,  standard  deviation  =  0.627,  range  =  3. 
Most  participants  (n  =  39;  92.9%)  indicated  the  prototype  was  of  extreme  relevance  to 
maintenance  operations.  One  participant  did  not  respond  to  statement  four. 
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(e)  Statement  five  asked  whether  prototype  concept  was  a  good  one.  The 
histogram  of  the  frequency  distribution  for  statement  five  is  presented  in  Figure  13.  The 
mean  was  4.71,  standard  deviation  =  0.427,  range  =  2.  All  participants  (n  =  42;  100%) 
indicated  the  concept  of  the  prototype  was  a  good  one.  One  participant  did  not  respond 
to  this  statement. 


"Is  a  good  concept" 


Strongly  Agree  Neutral  Disagree  Strongly 

A9ree  Disagree 


Figure  13:  Exit  Survey,  Part  II,  Statement  Five,  Response  Distribution 


(2)  Responses  to  Open-ended  Questions 

Part  III  of  the  exit  survey  contained  three  open-ended  questions  for  the 
participants  to  respond  to  their  overall  satisfaction  with  the  prototype.  Every  participant 
availed  himself  of  this  opportunity  to  provide  constructive  criticism.  The  responses  from 
all  43  participants  were  overwhelmingly  positive.  Eveiy  participant  indicated  there  was 
great  merit  in  a  tool  such  as  the  prototype  and  all  of  the  “criticisms”  were  presented  in  a 
professional/positive  manner.  The  desire  of  the  participants  was  to  take  this  prototype,  in 
its  current  form,  and  improve  it  for  their  use  in  the  fleet. 

(a)  Question  one  asked  the  participant  to  list  the  most  positive  aspects  of 
the  prototype.  Nine  participants  indicated  the  prototype  was  an  excellent  source  of  data 
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that  could  jbe  used  for  training,  trend  analysis,  and  decision  making.  Others  thought  the 
prototype  was  useful  to  provide  comparisons  between  variables  (aircraft,  mishap  type, 
location,  etc.).  Some  sample  inputs  include: 

•  " Recurring  maintenance  issues  stick  out  like  a  sore  thumb.  ’’ 

•  “MEIMS  provides  the  ability  to  determine  common  mishap  causal  factors  and 
prevent  future  ones  of  the  same  type.  ” 

•  "MEIMS  can  help  us  look  at  our  highest  risk  maintenance  working 

conditions  and  identify  those  areas  where  we  should  be  especially 
cognizant  of  potential  disaster.  ” 

(b)  Question  two  asked  for  the  most  negative  aspects  of  the  prototype. 
Overall  comments  indicated  that  participants  with  lower  than  normal  computer  “savvy” 
found  it  initially  more  difficult  to  navigate  and  understand  the  operation  of  the  prototype. 
However,  as  interaction  time  with  the  prototype  increased,  so  did  the  ease  of  operation. 
Problem  areas  of  the  prototype  application  were  focused  in  one  of  three  areas:  HFACS- 
ME  terminology,  interface,  and  data  entry. 

(1)  HFACS-ME.  Ten  participants  noted  the  HFACS-ME 
taxonomy  is  not  an  ingrained  part  of  everyday  terminology  and  thus  found  it  difficult  to 
understand.  The  ability  to  access  the  HFACS-ME  Code  definitions  from  various  parts  of 
the  prototype  helped,  but  additional  explanation  of  the  each  vice  a  mere  translation  of  the 
three-letter  acronym  would  have  added  more  value  to  the  participant.  The  participants 
felt  that  any  eventual  end-user  of  the  prototype  would  need  a  good  working  knowledge  of 
HFACS-ME  in  order  to  be  able  to  get  the  most  use  out  of  the  prototype;  and  that  even  the 
training  received  as  part  of  the  study  may  not  be  sufficient. 
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(2)  Interface.  Seven  participants  declared  the  prototype  to  not 
intuitively  obvious  in  its  operation.  The  fear  was  potential  end-users  with  a  lack  of 
general  computer  skills  would  be  discouraged  from  using  the  prototype.  However  two 
participants  commented  that  the  prototype  became  more  user  friendly  with  each  use.  Six 
participants  said  there  was  not  enough  on-line  help  available  for  usage  training. 

(3)  Data  Entry.  Though  there  were  only  four  negative  comments 
about  data  entry,  they  were  all  astute  observations  made  by  participants  who  obviously 
had  advanced  computer  skills  (though  they  were  indistinguishable  from  others  based  on 
their  demographic  inputs).  Comments  on  data  entry  included  not  having  positive  closure 
when  data  is  entered  and  being  able  to  enter  the  same  data  twice  with  no 

penalty/feedback.  The  remaining  two  comments  were  focused  on  unclear  procedures  for 
data  entry. 

(4)  Other  “negatives”. 

•  Navigation  issues  were  minor,  limited  to  suggestions  for  improved  access 
between  pages  (being  able  to  go  directly  from  one  page  to  another  without 
having  to  back  out  of  previously  selected  pages-four  participant  inputs). 

•  If  the  participant  selected  parameters  for  a  desired  function  (graphing  or 
report)  that  were  so  specific  that  no  data  matched  them,  the  function  appeared 
not  to  work.  The  “error”  message  displayed  to  the  participant  did  not 
satisfactorily  explain  the  problem. 

•  In  some  instances  the  three-dimensional  graphs  in  the  front  hide  data  in  the 
back.  Also,  the  graphs  did  not  fully  define  the  “colored”  categories. 
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•  Computer  quality.  There  were  several  instances  of  one  or  more  of  the 
functions,  especially  in  the  graphing  option,  not  working  due  to  lack  of 
memory  in  the  computer. 

(c)  Question  three  asked  for  suggested  changes  to  the  prototype.  The 
participants  brought  out  several  key  points  critical  for  inclusion  in  future  variations  of 
MEIMS.  Most  of  the  suggestions  related  directly  to  one  or  more  of  the  previously 
mentioned  “negatives.”  Nine  comments  were  made  about  improving  the  ability  for  the 
end-user  to  understand  HFACS-ME  through  either  improved  HFACS  definitions  within 
the  prototype,  additional  help/tutorial  on-line,  and  formal  training  for  all  end-users.  Eight 
participants  also  made  suggestions  to  improve  the  interface  and  navigation  of  the 
prototype  to  increase  usability  (e.g.,  adding  additional  methods  to  view  HFACS-ME 
definitions  and  better  descriptions  of  Levels  1,  2,  and  3) .  Though  no  specific  comments 
were  made  about  data  entry  improvement,  the  “negatives”  mentioned  above  imply  fixes 
to  be  made:  providing  positive  feedback  upon  entering  data,  not  being  able  to  enter  the 
same  data  more  than  once,  and  making  the  data  entry  procedures  simpler  and  more  clear. 

A  noteworthy  input  made  by  four  individual  participants  was  in  the  area  of  “target 
end-user.”  The  original  intent  of  MEIMS  was  for  it  to  be  distributed  to  squadrons  for  use 
in  both  data  retrieval  and  data  entry.  Four  participants  made  strong  statements  to  the 
effect  that  the  squadron  level  end-user  should  not  be  able  to  input  data  into  the  system, 
but  that  it  should  be  done  at  a  higher  level,  such  as  at  the  Naval  Safety  Center  where  the 
understanding  of  HFACS  and  HFACS-ME  is  greater  and  thus  so  is  the  ability  to  correctly 
input  data  into  MEIMS. 
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Other  inputs: 

•  Four  participants  indicated  their  desire  to  have  more  information  about 
each  mishap  available  for  viewing  (e.g.,  long  summary  vice  short 
summary,  adding  the  narrative  of  the  mishap,  etc.). 

•  Increasing  the  size  of  the  data  base  by  using  mishaps  prior  to  1989  and 
using  hazard  reports  was  felt  to  be  a  means  of  improving  the  quality  of  the 
data  (three  participants). 

•  Eight  specific  changes  to  the  actual  interface  were  also  suggested  (e.g., 
increasing  text  box  size  in  order  to  view  all  of  the  data  field,  a  better 
method  to  show  aircraft  model  to  prevent  confusion  by  adding  the 
nickname  to  the  model  number:  EA-6  Prowler,  E-6  Mercury;  being  able  to 
scroll  through  the  chronological  report  vice  viewing  it  page  by  page; 
separating  H-l  into  AH-1  and  UH-1  categories,  etc.). 

•  Using  a  higher  speed,  larger  memory,  improved  processor  computer  was 
also  suggested  to  improve  efficiency  of  MEIMS. 
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V.  SUMMARY,  CONCLUSIONS,  AND  RECOMMENDATIONS 

A.  SUMMARY 

Naval  Aviation  has  determined  to  reduce  its  mishap  rate.  The  reduction  of  human 
error  involved  in  maintenance  related  mishaps  will  be  one  step  in  achieving  that  goal; 
now  it  has  to  find  appropriate  tools  to  accomplish  this.  The  Human  Factors  Analysis  and 
Classification  System-Maintenance  Extension  (HFACS-ME)  is  a  taxonomy  which  covers 
maintenance  operations  and  falls  in  line  with  the  Naval  Aviation  Safety  Program’s  notion 
of  multiple  causal  factors,  the  idea  of  sequential  events  leading  to  an  event,  and  several 
established  human  factors  theories.  HFACS-ME  has  been  successfully  used  to  examine 
human  error  in  mishaps  and  incidents.  The  prototype  MEIMS  (Maintenance  Extension 
Information  Management  System)  tool  is  a  safety  information  management  system  based 
on  the  HFACS-ME  taxonomy  used  to  facilitate  the  characterization  and  analysis  of 
human  error  in  Naval  Aviation  maintenance  mishaps.  Tools  such  as  a  final  version  of 
MEIMS  will  provide  assistance  in  identifying  human  error  patterns  and  facilitate 
intervention  development. 

B.  CONCLUSIONS 

The  participants’  overall  satisfaction  of  the  prototype  MEIMS  tool  indicated  there 
is  a  need  to  provide  access  to  mishap  data  information  for  use  in  training,  analysis,  and 
investigations.  Participant  feedback  demonstrated  the  concept  of  MEIMS  to  be  sound 
and  its  tie-in  with  Maintenance  Operations  readily  apparent.  However,  the  prototype 
requires  some  adjustment  before  successful  implementation  by  end-users  can  be 
achieved. 
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The  prototype,  itself,  received  slightly  lower,  though  still  very  positive,  ratings 
than  the  concept  of  a  maintenance  human  error  related  information  management  system. 
Even  though  a  tool  can  be  of  good  intent,  if  it  is  not  considered  usable  by  the  end-user,  it 
will  sit  on  the  shelf.  For  MEIMS  to  be  “the  tool”  it  must  have  its  shortcoming  resolved: 

•  Lack  of  general  maintenance  organization  HFACS-ME  training  including 
familiarity  with  HFACS-ME  terminology.. 

•  Less  than  desirable  human-computer  interface  for  end-users  with  below 
average  computer  skills. 

•  Poor  data  entry  confirmation  indications.  The  inability  for  the  end-user  to 
enter  data  into  the  prototype  in  a  simple  and  consistent  format  may  lead  to 
inconsistent  inputs  and  hence  poor  data  (i.e.,  garbage  in,  garbage  out). 

•  Also  several  minor  shortfalls  need  to  be  refined: 

•  Lack  of  standardized  and  convenient  navigation. 

•  Poor  “error”  messages  in  the  cases  of  null  data  selections. 

•  Some  three-dimensional  graphs  hiding  data  depending  on  the  view  selected. 

•  The  inability  to  run  the  prototype  successfully  on  some  older  personal 
computers. 

Providing  solutions  to  these  identified  failings  will  improve  the  usability  of  future 
versions  of  MEIMS;  and  subsequently  the  opportunity  for  it  to  be  a  factor  in  reducing  the 
aviation  mishap  rate  is  enhanced. 
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C.  RECOMMENDATIONS 


(1)  Recommended  Prototype  MEIMS  Tool  Improvements 

•  HFACS-ME.  Incorporate  improved  HFACS-ME  definitions  within 
MEIMS  by  ensuring  access  to  the  definition  page  is  available  on  every 
form.  Better  descriptions  of  the  HFACS-ME  acronyms  would  also 
improve  usability  and  understanding.  Incorporating  additional  on-line 
help/tutorials  will  improve  the  end-users  knowledge  of  HFACS-ME  and 
make  MEIMS  a  more  productive  tool  for  their  use. 

•  Interface.  A  computer  science  expert  should  participate  in  the  fine  tuning 
of  MEIMS  interface  options  to  ensure  navigation  is  consistent  and  easily 
done  for  those  with  sub-par  computer  skills. 

•  Data  Entry.  Ensure  data  entry  procedures  are  made  as  simple  and  clear  as 
possible  including  providing  positive  feedback  to  the  end-user  once  the 
entry  has  been  taken.  MEIMS  must  not  allow  repeat  entries  of  the  same 
data. 

•  Target  End-user.  Unless  data  entry  procedures  can  be  significantly 
simplified  MEIMS  should  be  used  as  at  the  maintenance  organization 
level  in  the  read/analysis  mode  only.  Entry  of  data  should  be  conducted 
by  higher  levels  (i.e.,  the  Naval  Safety  Center  for  Naval  Aviation)  where 
the  understanding  of  HFACS-ME  is  greater  and  thus  so  is  the  ability  to 
correctly  input  data  into  MEIMS. 

•  Include  a  longer  summary/narrative  for  each  mishap. 
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•  Additional  data.  Include  mishaps  prior  to  1989  and  all  hazard  reports  to 
improve  the  quality  of  the  data  base. 

•  Increase  text  box  sizes  to  view  all  data  field. 

•  Change  aircraft  identifier  to  include  aircraft  nickname  in  addition  to 
type/model  to  avoid  similar  names. 

•  Separate  AH-1  and  UH-1  into  two  categories,  vice  only  H-l  due  to  the 
aircraft’s  inherent  differences. 

•  Change  the  chronological  report  to  a  scrolling  view,  vice  page  by  page  to 
improve  readability. 

•  If  a  selection  is  made  for  data  that  has  a  null  value,  ensure  the  error 
message  indicates  the  lack  of  response  from  MEIMS  is  due  to  “no  data 
available  for  selected  entry”  vice  simply  an  error  with  the  system. 

•  Arrange  data  on  three-dimensional  graphs  so  that  the  fields  with  the 
largest  numbers  are  put  in  the  rear  rows  and  scaled  down  to  the  front  so 
that  no  data  is  hidden  to  the  end-user. 

•  Suggested  Computer  Capability.  Ensure  end-users  understand  that 
computers  with  higher  speed  processors  and  larger  memories  will  improve 
the  efficiency  of  MEIMS. 

•  Add  an  option  to  include  percentages  on  the  three-dimensional  graphing 
function,  vice  only  quantity.  This  will  show  relative  weight,  vice  always 
being  more  heavily  weighted  for  aircraft  types  with  a  larger  inventory 
(FA-18,  H-46,  etc.). 
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(2)  The  Future  of  MEIMS.  The  research  for  this  paper  involved  data  derived 
from  the  Naval  Safety  Center’s  data  base  of  Naval  Aviation  mishaps.  A  variation  of  the 
prototype  MEIMS  tool  could  be  revised  to  include  data  from  commercial  mishaps  (both 
passenger  carriers  and  general  aviation).  Civilian  aviation  also  has  a  record  of  human 
error,  including  maintenance  related  human  error,  contributing  to  mishaps. 
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APPENDIX  A 


PROTOTYPE  MAINTENANCE  ERROR  INFORMATION  MANAGEMENT 
SYSTEM  (MEIMS)  TOOL  REVIEW 

1.  MAIN  MENU 

The  MainM^iuappeajs^fterMEIMSJCONis_selected  (see  Figure  Al) 


Mainftinaocc  Errors'  ,  I  : a \ i 


ISs**  »* 


FigjwtfAl:  Prototype  MEIMS  Tool  Main  Menu 

Select  “Query  Menu”  command  button  to  view  Query  Menu  (see  Figure  A2). 

2.  QUERY  MENU 


Queries  | 


Aircraft  Model 

F-14,  H-46... 

Aircraft  Type 

TACAIR,  Hefo... 

Branch  of  Service 

USN/USMC 

Location  of  Mishap 

Slf  ^ 

Mishap  Class 

A.B.C 

Mishap  Type 

FM,  FRM,  AGM 

;  3fear  of  Mishap  ;|||| 


Multiple  Criteria ;  H 

i  \ .  ‘  f 

; Icmenortfhimtheieft 


HFACS-ME  Summary 

!|  ^;teTat8gory:usmga|S:| 
Multiple  Criteria  Qtiery^gJ: 

Return  to  Main  Menu 


/  Figure  A2:  Query  Menu 

Select  “Aircraft  Model”  command  button. 

The  Query  by  Aircraft  Model  menu  appears  (see  Figure  A3). 


|§  Query  by  Aircraft  Model 


Select  Aircraft  Model 


<  Back 


View  Selection 


Figure  A3:  Quej^by  Aircraft  Model  Sub-MenuN^ 

Select  aircraft  model  in  combo  box  and  select  “View  Selection”  commanabutton. 


The  “Summary  of  Mishap”  form  appears  (see  Figure  A4). 


I  §=]  Summary  o  f  Mishap 


Maintenance  Mishap  Query 

Mishap  Number  f  2 


DaleofMshap  |1/28/S4 

AJrc^Type  r  ?  .  jFvTj.-,. 

|  Mishap  typo  |fm  :  • . 

Brief  Description 

;  :  |Adt  on  FCF  hod  left  eng  oil  hot  light  during  dim 


Contributing  Factors 


Qes*  ol  Mishap .  |c"~ 


Branch  of  Service  pJSN 


<Atfraait  Category  -  Jtacajr 


location  of  Mishap  Inshore 


II  Define  HFaCS  Codes  II 


LeveM  Level  2  Level  3 


:  Work  dr  supv.  end  COI  foiled  to  provide  adequate  tech  data  and  procedures  due  to  complecen  SC  SON  IDO 


Record:  U I  ±\\  T  ►  I  H  of  66  CFUtaecQ 


Figure  A4:  Sumjutffy  of  Mishap  Form  with  F14  Selected  as  Aircraft  Type 

Select  the  right  arrow  after  “Record:”  on  the  bottom  line  to  view  additional  records. 


F14  record  number  two  appears  (see  Figure  A5).  Note  that  if  the  mouse  arrow  is  placed 
over  a  text  box  or  other  control,  a  control  tip  text  appears.  — l  Select  “<Back”  to  return. 


I  §5]  Summary  or  Mishap 


Maintenance  Mishap  Query 

M«hap  Number  P”  Dess  of  Mishap  |c” 


DateofMishap  {3/23/34  Br 

AircraftType  jFM  Ai 

Mishap  Type  .  |aGM  Lc 

Brief  Description 

lAJM^^C  released  from  shoulder  weapons  rail  during 


Branch  ofServee  U! 


; Mishap  Class:  A.  B.  C] 


Aircraft  Category  JTACAIR 

Location  of  Mishap  jAshore 


Contributing  Factors  I - — — - 


j  IMA  Supervisor  Failed  to  Follow  TECH  Procedures  Monitor  Mnjrrtenance  Actions 


Level  1  Level  2  Level  3  1 


SC  SON  MIS 


Record:  H{  4  |j  2  i  |  M  !► 


Figure  A5:  Summary  of  Mishap  Form  with  F14  Selected  as  Aircraft  Type, 

Record  Two 
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Query  Menu 


^Queries- 1.>; 


;||  jgS 

Aircraft  Type'  . '  *  -  ‘  ::. 

**•  *  * 


Multiple  Catena  1 


Astefe»Einfeted„  j  | 

Zl 


1  A,ac:->f^ 

j/Type  sfij  m 

B^fRM/AGM 

r«*ref  Miship  Alv 

^•^jEaifcaAy  "" *  v  '  ■  ■ *  -  a  '  - 1 


£1® 


:--A  : 

Muitipte  Criteria  Query 

Return  to  Main  Menu 

.;„  .  .  .  .  '  : ' _ w 


Additional  queries  may  be 
executed  by  selecting  any  of  the 
control  buttons  on  the  left  of  the 
Query  Form  (see  Figure  A6). 


Query  by  Aircraft  Type 


Select  Aircraft  Type; 


Select  by  Location 


■%  Select  Mishap  Location 

i  . . . $2  :i  T''M  *- 

:  »  „  -s  }'•  V~  v 


TACAIR 
HW 
I  Trainer 


Ashore 


Ashore 


Select  By  Service 


Select  Service 

" 

'  I 


^IMBIBll 


ifias  i.L ,  <  Back  ■  ~  1  ~ 

M;.  -  - . . * 


USN 

USMC 


gpjgl 


Select  by  Class 


'x ■-.  nwi;  j.  r  - , 

“is . 


<'h;  ‘  t"  _ 

fl j  Embarked  | 

-  '  <BoA”  |  /  :.  -  | Detached  Vfi>  <Back  I  -  View 

■  J  *-ri;A  ..  H Unknown  .  ’ -  \:;f '■ 


Select  Mishap  Class 

-  ;  -  »  ,  v 


.  I 


itlllsi 


Unknown 


Select  by  Type 


’■  :  |  M’  &,%■  :  ■  :  :-:i:  ri :  r>"  '  :'L:  . 

■i^^'M^Type  ™ 

' r ; , :  *» ( A; 


WFM 


Hjc 


Select  by  Year 


i  -Select  Cal en dar -  Year :  FllE^lS " !Yf' ? " m ; " ; : 


llltllii! 


|jll|  :<:6aefc  jj||||j|  Jpll 


j 


1998 
1997 
1 QQK 


..Jtill 

■'  K  ??S;f  ' 


on 


Figure  A6:  Query  Menu  with  Sub-Menus  for  Additional  Queries 

Select  Multiple  Criteria  command  button  on  Query  Menu  to  more  precisely  define 
query  (see  Figure  A7). 

'Quttri»r|'  '  *  ’  ’  "A  r  „  &  -  ««v  >  v  *  T  ~  i ,  w‘  *?  s : 

- 1  -  •  * . . . .  -  *  " 1 '**  *  *  • . "*  ^ 


J  A  Model 

•  :  ,/M 


□  Multiple  Crftena 
uriterw^fnstn  deleft 

rjj 


Figure  A7:  Query  Menu  with  Multiple  Criteria  Sub-Menu 
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Define  query  by  selecting  desired  check  boxes  and  detailing  information  in  combo  boxes 
(see  Figure  A8).  / 

Select  Multirfe  Criteria  Query  I  / 


Click  the  box 
nexttothe 
criterion  that  you 
wish  to  select 

Select  the 
desired  item 
from  the  drop 
down  box.  i 


.ft  Aircraft  Model 
r  lAircraft  Type 
Kjtr:.  branch  of  Service 
-ocation  of  Misha 
Mishap  Class  ! 
r  /Mishap  Type  ; 
r  /  Year  of  Mishap 


(Embarked  I 


^View  Selection 


_ Eigwe-ASr  Multiple  Criteria  Sub-Menu  with  Desired  Selections 

Select  “View  Selection”  to  view  Summary  of  Mishap  form  (see  Figure  A9). 


|  §1)  Summary  of  Mishap 


t  Maintenance  Mishap  Qtieiy 

. .  Mbftep  Nombwr  . .  fs?U  7 


Xrat*T&e  :  /  pfi 


tel:  F-14.H-46..K 


[A/C  DEP  CTRL  FLT  During  CV  Brs«c 

Cortribmbg  Factor*  \ 

NASCMAJNT  INST  Does  not  REO  Solely  E&U1P 
MAINT  Men  Lost  SA  While  Performing  MAINT^ 


[QbHiw  HFA£SCode*l 


Uv*li  Lavel2  Uvai3 

“SC  ORG  DOC 

:MC  MED  MNT 


«*card:  H  QJj "  1  ►  I  ►!>*{<*  4  <Tii»r«d) 


Figure  A9:  Summary  of  Mishap  Form  from  Multiple  Criteria  Selection. 


Note  that  the  desired  selection  appears  in  a  blue  text  box.  Select  “Define  HFACS  Codes’ 
command  button  to  define  Level  1,  2,  and  3  codes  (see  Figure  A 10). 
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Organizational  I  Hazardous  Inadequate  Doc-  Inadequate  Inadequate  |  Inadequate 

{°RG)  |  Operations  (HAZ)  urnentatton  (DOC)  Design  (DES)  Resources  (RES)  |  Processes  (PRO) 

Supervisory 

Conditions  (SQ  Squadron  (5QN)  ■  Inadequate  Inappropriate  Uncorrected  Supervisory 

H|M| _  [  Supervision  (1DQ)  Operations  (OPS)  Problem  (PRB)  Misconduct  (MIS) 


Medical  (MED)  ■  Mental  Stale  (MNT)  Physical  Limitation  (LI M) 

■  State  (PHY) 


Maintainer 
Conditions  (MC) 

Crew 

Coordination 

(CRW) 

Readiness  (RDY)  1 

1  Communication 

Assertiveness 

Adaptability/  I 

[  (COM) 

(ASS) 

Flexibility  (ADA)  1 

Training/  Certification/  Infringement  (INF) 

Preparation  (TRG)  Qualification  (CRT; 


[[Close  Forml 


Environment  I 
(ENV)  ] 

|  Lighting/ 

1  Light  (LGT) 

Weather/ 
Exposure  (WXE) 

Environmental  1 
Hazards  (EHZ)  1 

Working 
Conditions  (WQ 

Equipment  (EQP)  1 

1  Damaged  (DMG) 

Unavailable  (UNA) 

Dated/  1 

Uncertified  (DUC)  1 

Workspace  (WRK)B 

1  Confining  (CON) 

Obstructed  (OBS) 

Inaccessible  (IN A)  1 

Error  (ERR) 

Maintainer  _ 


Violation  (Vi 0) 


Attention  (ATT) 

Memory  (MEM) 

Knowledge/ 
Rule-Based  (KNW) 

Skill  Based  (SKL) 

Judgement/ 

Decision-Making 

{ rptr.\ 

Routine  (ROU) 

Infraction  (IFC) 

Flagrant  (FLG)  j 

Sabotage  (SAB) 

Figure  A10:  “Define  HFACS  Codes”  Form 


This  form  may  be  selected  at  various  points  throughout  the  prototype  in  order  to  receive 
HFACS  code  definition.  Return  to  Query  Menu  (see  Figure  A1 1). 


Select  HFACS-ME  Summary  command  button. 
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The  HFACS-ME  Summary  form  appears  and  displays  HFACS  level  1,  2,  and  3  summary 
for  all  aircraft.  Refine  query  by  selecting  desired  information  in  combo  boxes  (see  Figure 
A12).  \ 


I Maintenance  Mishap  Databse  -  [All  Mishaps  With  Factors/  Codes]  15WElRi| 

■  Inadequate  Doc-  I 

umentation  (DOC)  I 


|  Eil«  Edit  Josert  Records  rSfmdcfw  Help 


Hazardous  ■  Inadequate  Doc-  ■  Inadequate  ■  Inadequate 

Operations  (HAZ)  I  umentation  (DOC)  I  Design  (DES)  I  Resources  (RES) 


5  23%  2  9% 


Inacequate  I  Inappropriate  I  Unconected  ■  Supervisory 

Supervision  (IDO)  I  Operations  (OPS)  I  Problem  (PRB)  I  Misconduct  (MIS) 


Environment 

Liqhtmg/ 

1  Weather/  H 

environmental 

(ENV) 

Light  (LOT) 

|  Exposure  (WXE)  | 

Hazards  (EHZ) 

T otal  Number  of  HF  Mishaps 

22 


Working 
|  Conditions  (WQ 


Equipment  (EQP) 


Damaged  (DMG)  |  Unavailable  (UNA;  ^  Dated/ 

Uncertified  (DUC) 


0  0%  0  0%  0  0% 


lip&t  mishap -class;  A,  B,  C.  "  •  . :  ;  ~  '  1  '  '  ~~  ~ — 

Figure  Al2:  HFACS-ME  Summary  Form 
Return  to  Main  Menu  (see  Figure  A13). 

3.  REPORT  MENU 


■  PM its^An  'X*k> 


■  IniormatiouManagmentg^ 


f  System 

Select  “Report  Menu”.  Report  Menu  appears  (see  Figure  A 14). 


1  §1] Report  Menu 

Report*  | 

■* :  gjas  -  i“  k  :  stjaas » «»&  *»■«**&  * 

2|®k;  ■  Select  s  Report  -  *  v  " 
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|jl§ 

>  - 

1 
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iSSHfi 

J  MishaprDistrihution  -  Service  ; 

§||J;;f|j§V 

*'  ' 

Jll  Mishap  attribution  -  location  . 

U;0M 

?tS9S»! Ustmg  \ 

■  ■ 

fi;l :;;§f  - i;--: ■  ■  ■ : ivi.-:  "V  VT:p  * 

«ll|f 

.  ||| 

Figure  A14:  Report  Menu 

Select  “Mishap  Distribution- All  Mishaps”  command  button  to  view  corresponding  report 
(see  Figure  A1 5). 
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Figure  A15:  Mishap  Distribution-All  Mishaps  Report 
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Additional  reports  may  be  selected  from  the  Report  Menu  including  “All  Mishaps- 
Chronological  Listing”  (see  Figure  A16). _ 


Chronological  listing  of  all  Maintenance  Related 
Mishaps  by  Aircraft  Type  (1989-1999) 


Description 


FRM  TACTS  podd«ptrt*d  LAU-7  g\*d»d  mualt  Uuncfa«t  4 

FM  AC  onPMCF,  ENO  Flea  «d  Out 

AO  W  During  groutdUxi.  »efl  crocMdovw 

FM  Ctnopr  lo* 

FM  Aift  1  aoUd  non  gwd  up.  k>cCa  B  *0  dtxm  «cd !  ocfc* 

AO W  EODTMMMambwisjuradwhsluogflvi  igntodtf 

FM  A/C  DEP  CoatfoOrfFLT  During  FCF  SLAT  Ck»cku^Wa^  DMO 

AQM  UNCOMMANDED  ENGINE  ACCELERATION  CAUSED  ACFT  TO  STRIKE  ANOTHER  AC  FT 
FM  A/C  DEP  RWY  oc  LDO  tc  Rolled  Ov*t 

FM  AC  C«ugHFu.<nT/ORoD.PaotSuce»W\jUyAbcru<lTJO 

FM  ENOFirtDuttoFu)ILMfc 

FM  AC  Cr*4»4uioDt«trt  After  Lon  of  ENO 

FM  CANOPY  DEPARTED  AIRCRAFT  DURJ NO  FLIOHT 

AOM  AC  Brdo  Loo*» Duong Hj^vPo«r«T String  BLIXJ 

FM  AIRCRAFT  DEPARTED  RUNWAY  AFTER  SLOW  LANDINO 


Figure  A16:  All  Mishaps-Chronological  Listing  Report 

Return  to  Main  Menu  (see  Figure  A17). 


4.  EXPERT  GRAPH  MENU 


|  System  (MEIMS) 


0 


|  Report  Menu.%; 
'iT^Expert  Graph  Menu: 

e»t 


___ — — ""  Figure  A17:  Main  Menu 

Select  “Expert  Graph  Menu”  (see  Figure  A18). 


Add  New 


r  Figure  A18:  Expert  Graph  Menu  and  X-Axis  Cate  ;ory  Sub  Menu 

Select  categories  for  X  and  Y  axes.  Make  selections  for  X  axis! 
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Data  Entry  Form  appears  (see  Figure  A22). 


Figure  A22:  Data  Entry  Form 

Enter  data  for  new  mishap  (see  Figure  A23).  Select  “Add  New  Factor”  for  each  factor  to 
enter.  When  3  Level  Code  entered,  2ncl  and  1st  Level  Codes  are  automatically  entered  by 
the  prototype.  x\ 


y/Y\  '  ‘  •*  •'*.*.*  *  -  [Record:  'Hf '4  If  T  >  I M  (►*lof  i 

fteqyd:  t<\<\\  i  ►  1  h  !►*!  of  1  ^ 

Figure  A23:  Data  Entry  Form  with  Sample  Data  Applied 

Once  complete,  select  “Close  Form”.  Final  Data  Entry  Form  appears  (see  Figure  A24). 


Click  on  "Enter"  box  to  complete  data 
entry.  Wait  for  check  in  box  to  appear 
before  selecting  "Close  Form". 

VfzEz  O  Done 


Close  Form 


Figure  A24:  Final  Data  Entry  Form. 

Select  “Enter”  to  complete  data  entry.  After  check  appears  in  “Done”  box,  select  “Close 
Form”. 


APPENDIX  B 


PROTOTYPE  MAINTENANCE  ERROR  INFORMATION  MANAGEMENT 
SYSTEM  (MEIMS)  TOOL  EVALUATION 

Background.  Thank  you  for  participating  in  a  usability  study  (evaluation)  of  a 
prototype  tool  for  the  Maintenance  Error  Information  Management  System  (MEIMS). 
This  tool  was  developed  by  CDR  Brian  Wood,  USN  as  part  of  a  thesis  project  for  his 
Master  of  Science  program  in  Information  Technology  Management.  The  information 
management  system  was  developed  to  effectively  address  and  identify  patterns  of  human 
error  in  Naval  Aviation  maintenance-related  aircraft  mishaps.  The  Human  Factors 
Analysis  and  Classification  System  Maintenance  Extension  (HFACS-ME)  taxonomy  is 
the  foundation  of  MEIMS  and  is  an  effective  method  for  classifying  and  analyzing  the 
presence  of  human  error  in  maintenance  operations  leading  to  major  mishaps,  accidents 
of  lesser  severity,  incidents  and  maintenance  related  personal  injury  cases.  However, 
working  with  a  large  database  (approximately  600  Naval  Aviation  maintenance-related 
mishaps  in  Fiscal  Years  90-99)  is  very  labor  intensive.  Given  the  capability  of  current 
relevant  database  tools,  an  improved  information  management  system  will  bring  HFACS- 
ME  to  the  next  level. 

MEIMS  captures  maintenance  error  data,  facilitates  the  identification  of  common 
maintenance  errors  and  associated  trends,  and  supports  understanding  of  how  to  identify 
human  errors  in  the  future.  The  target  audience  for  this  information  management  system 
tool  includes  safety  personnel  (data  entry  &  retrieval  by  unit  safety  officers,  other  safety 
&  training  personnel,  maintenance  officers,  maintenance  supervisors),  mishap 
investigators-for  data  retrieval  (Aircraft  Mishap  Board  members,  squadron  safety 
officers),  and  analysts  (from  the  Naval  Safety  Center,  the  command’s  safety  officer  or 
one  from  its  higher  headquarters).  A  usability  study  demonstrated  the  effectiveness  of 
the  tool.  This  tool  allows  can  directly  lead  to  a  decreased  mishap  rate  and  overall 
increased  mission  readiness  due  to  the  training  and  analysis  opportunity  it  provides. 

Usability  Study.  You  will  be  given  a  packet  of  instructions  to  guide  you  through 
MEIMS.  You  will  be  asked  to  make  comments  on  the  effectiveness  and  usability  of  the 
prototype  system  during  your  testing  phase.  Additionally,  you  will  be  asked  to  complete 
an  “exit  survey’  after  completion  of  your  testing.  Questions  will  include  demographic 
information,  objective  questions  about  MEIMS  usability,  and  subjective  questions  and 
comments  for  areas  not  covered  in  the  objective  section.  The  study  should  take  no  more 
than  15-20  minutes. 

Completion  of  Study.  Upon  completion  of  your  testing  and  survey  you  will  be 
asked  to  return  your  packet  of  instructions  to  CDR  Wood’s  office  (E-305,  East  Wing 
Herrmann  Hall).  Pull  this  cover  sheet  off  and  put  in  the  box  marked  “Cover  Sheet.”  Put 
the  remainder  of  the  survey  (ensure  stapled)  in  box  marked  “Remainder  of  Survey.” 

Thank  you  again, 

Brian  Wood 
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Instructions  for  Prototype  Maintenance  Error  Information  Management  System 

(MEIMS)  Tool  Evaluation 


Start-up 

1.  Go  to  room  E-300  (Computer  Lab),  E-320  (Ready  Room),  or  E-322  (Computer  Lab). 
Turn  on  computer  (does  not  need  to  be  logged  into  NPS  LAN). 

Question  1:  What  is  the  name  of  your  computer  (aircraft  name  on  CPU)? 


2.  When  Log-in  menu  appears,  select  <ESC>  (this  bypasses  log-in  requirement). 

3.  When  Desktop  (main  Icon  screen)  appears,  double  click  (clicks  are  always  with  left 
mouse,  unless  otherwise  stated)  on  “MEIMS”  Icon.  This  will  start  the  MEIMS 
application  (in  Microsoft  Access  97). 

Main  Menu 

4.  You  will  now  have  the  Main  Menu  displayed  with  the  world  famous  Supersonic 
Hornet  photo  in  the  background. 

5.  Note  the  five  categories  next  to  the  command  buttons  on  the  bottom  right  portion  of 
the  screen.  The  system  has  “focus”  on  “Query  Menu”.  Note  the  information  on  this 
button  in  the  bottom  left  gray  buffer  above  the  Windows  Start  button.  Place  the  mouse 
pointer  over  the  Query  Menu  box  (don’t  click,  if  you  do,  select  <Back>  on  subsequent 
page)  and  note  information  that  appears  in  the  Text  Box  (both  of  these  sources  of 
information  will  be  available  throughout  MEIMS). 

6.  Select  <Tab>  and  view  the  same  information  for  the  remaining  four  command  buttons 
(note,  if  you  select  <Exit>  you  will  have  to  re-enter  the  system  (see  step  3  above). 

Question  2:  Is  the  terminology  clear  enough  to  understand  what  each  of  the  four 
command  buttons  does?  If  not,  what  could  be  changed  to  make  it  clearer? 


7.  Select  (click  or  tab  to  &  enter)  <Queiy  Menu> 


Query  Menu 

8.  Note  there  are  two  sections  on  the  Query  Menu.  The  left  half  of  the  screen  has  seven 

categories  to  help  you  define  how  you  would  like  to  view  the  mishap  data.  The  right  half 
of  the  screen  has  four  command  buttons.  c 

9.  Select  <Aircraft  Model> 
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10.  Another  form  appears:  “Query  by  Aircraft  Model”.  Select  your  type  aircraft,  then 

select  <View  Selections  “Summary  of  Mishap”  Form  appears.  Note,  your  aircraft 
selection  has  a  blue  background.  Review  the  “Brief  Description”  of  the  mishap  and  the 
“Contributing  Factors.”  View 

Question  3:  What  aircraft  did  you  select? _ 

How  many  separate  mishaps  of  that  type  aircraft  are  in  the  database? _ 

View  one  of  the  mishaps. 

What  are  the  level  3  codes  &  what  do  they  mean? 

How  did  you  find  that  info? _ _ 

When  you  are  through  viewing  the  data,  select  <Close  Form> 

Select  another  aircraft  model  (optional). 

When  complete  select  <Back>  on  Query  by  Aircraft  Model  Form 

1 1 .  Select  another  category  (your  option)  &  view  the  data. 

Question  4:  Which  (if  any)  of  the  seven  categories  do  you  find  useful? 


Which  (if  any)  of  the  seven  categories  do  you  not  find  useful? 


12.  Select  <Multiple  CriteriaS  Create  your  own  query  using  two  or  more  criteria. 
Question  5.  Did  you  find  this  function  useful?  Why  or  why  not? 


13.  Return  to  Query  Menu.  Select  <HFACS-ME  Elements^ 
Question  6:  How  many  total  mishaps  are  in  the  database? 


How  many  mishaps  have  a  level  one  category  of  Maintainer  Conditions? 
How  many  mishaps  have  a  level  two  category  of  Violations? _ 


14.  Return  to  Query  Menu.  Select  <HFACS-ME  Summary>. 
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Question  7:  How  many  total  mishaps  are  in  the  database? _ 

How  many  mishaps  have  a  level  one  category  ofMaintainer  Conditions? _ 

How  many  mishaps  have  a  level  two  category  of  Violations? _ 

Further  define  the  system  by  your  aircraft  model  (or  select  another  type). 

Question  8:  What  aircraft  did  you  select? _ 

How  many  separate  mishaps  of  that  type  aircraft  are  in  the  database? _ 

Conduct  further  queries  as  desired.  When  complete,  return  to  Query  Menu  &  return  to 
Main  Menu. 


15.  Select  <Report  Menu>.  Review  the  six  command  buttons  and  their  functions. 

16.  Select  <Mishap  Distribution  -  Aircraft>.  Find  your  type  aircraft  (or  review  another) 
in  the  report.  <Close>  the  report  &  return  to  the  Report  Menu. 

1 7.  Select  <A11  Mishaps-Chronological  Listing>.  Review  data.  <Close>  the  report  when 
complete  &  return  to  Report  Menu.  Return  to  Main  Menu. 

18.  Select  <Expert  Graph  Menu>.  Follow  directions.  Create  one  graph  with  aircraft 
model  (yours  and  1  or  2  others)  on  the  X-Axis  and  HFACS-ME  Level  One  (all  four 
codes)  on  the  Y-Axis. 

Question  9:  What  aircraft  did  you  select? _ _ 

Did  you  notice  a  difference  in  the  level  one  codes  between  the  aircraft  (if  so,  what)? 


Return  to  <Expert  Graph  Menu>.  Try  more  graphs  as  desired.  When  complete,  return  to 
Main  Menu. 

19.  Select  <Add  New  Data>.  Enter  the  following  three  mishaps  to  the  database: 
Question  10:  What  is  the  Mishap  Numbers  for  the  data  you  are  entering? 


Check  to  see  if  your  entries  were  added  to  the  database  by  Looking  at  the  end  of  the 
Chronological  Listing  on  the  Report  Menu  (look  for  your  Mishap  Numbers). 
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Question  11:  Did  you  see  your  data  in  the  Chronological  Listing? 
Return  to  Main  Menu  &  Exit  the  Program. 

20.  Please  fill  out  the  Exit  Survey  Questionnaire. 
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APPENDIX  C 


PROTOTYPE  MAINTENANCE  ERROR  INFORMATION  MANAGEMENT 
SYSTEM  (MEIMS)  TOOL  EXIT  SURVEY 

User’s 


Purpose:  This  survey  evaluates  a  user’s  overall  satisfaction  of  the  Maintenance  Error 
Information  Management  System  (MEIMS)  prototype  tool.  It  consists  of  three  parts. 

Part  I:  Demographic  Information.  Part  I  provides  the  user’s  aviation 
background,  computer  experience,  and  availability  of  software  and  hardware  systems 
used  in  the  Navy  and  Marine  Corps. 

t>  tt  /“T  User  Satisfaction  with  the  Four  Sections  of  the  MEIMS  Prototype  Tool 

Part  E  deals  directly  with  user  feedback  as  they  use  the  prototype  tool. 

Part  III:  User  Overall  Satisfaction  with  the  MEIMS  Prototype  Tool.  Part  m 
allows  users  to  give  general  feedback  about  the  prototype  tool. 


Impression  of  the  Maintenance  Error  Information  Management  System  (MEIMS) 

Prototype  Tool 


Part  I.  Demographic  Information 

Follow  the  instructions  after  each  numbered  question  or  statement. 


1 .  I  am  attached  to  a  command  that  primarily  performs  maintenance  (military  and/or 
civilian)  at  the: 

(Select  one  from  the  list  and  check  the  box) 

□  Squadron  Level 

□  Intermediate  Level  (AIMD) 

□  Depot  Level  (NADEP) 

□  Command  does  not  perform  aircraft  maintenance 

□  Other  (describe  if  other) _ 

2.  How  long  have  you  been  using  a  computer? 

(Select  one  from  the  list  and  check  the  box) 

□  Less  than  one  month 

□  One  month  to  less  than  one  year 

□  One  year  to  less  than  two  years 

□  Two  years  or  more 
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3.  What  software  do  you  normally  use? 

(Check  all  boxes  that  apply) 

□  Microsoft  Office  (Word,  PowerPoint,  Excel,  Access) 
What  version? 

(Check  all  boxes  that  apply) 

□  97 

□  2000 

□  not  sure  of  version 

□  other  (describe  if  other) _ 

□  Lotus  Smart  Suite  (Word  Pro,  Lotus  123...) 

What  version? 

(Check  all  boxes  that  apply) 

□  97 

□  9.5 

□  not  sure  of  version 

□  other  (describe  if  other) _ 

□  Corel  Word  Perfect  Office  (Word  Perfect,  Quattro  Pro...) 
What  version? 

(Check  all  boxes  that  apply) 

□  Corel  Office  7 

□  2000 

□  not  sure  of  version 

□  other  (describe  if  other) _ 

□  Other  (describe  if  other)  _ _ 

4.  What  software  application  categories  are  you  familiar  with? 
(Check  all  boxes  that  apply) 

□  Word  Processing  (MS  Word,  Word  Perfect,  Word  Pro...) 

□  Spreadsheet  (Excel,  Lotus  123,  Quattro  Pro...) 

□  Presentations  (PowerPoint,  Harvard  Graphics...) 

□  Graphic  Software  (Corel  Draw,  Adobe  Photoshop...) 

□  E-Mail  (Outlook,  Eudora,  AOL...) 

□  Database  (Access,  DBase...) 
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5.  What  computer  operating  systems  do  you  use? 
(Check  all  boxes  that  apply) 

□  Windows  (3.1, 95,  98,  2000) 

□  Windows  NT 

□  Macintosh 

□  UNIX 

□  Linux 

□  Other  (describe  if  other) _ 


Part  II.  User  Satisfaction  with  the  Four  Sections  of  the  MEIMS  Prototype  Tool 

Select  the  category  that  best  matches  your  impression  of  each  of  the  below  categories 
(and  check  the  box).  ° 

Strongly  Agree  Neutral  Disagree  Strongly 
Agree  Disagree 

I  feel  the  information  □  □  n  n 

on  the  MEIMS  tool  was  U  □ 

in  a  logical  form 


(comments) 


I  found  the  MEIMS 
tool  easy  to  navigate 

□ 

□ 

□ 

□ 

□ 

(comments) 

My  tour  of  the  MEIMS 
tool  was  very  interesting 

□ 

□ 

□ 

□ 

□ 

(comments) 

The  information  presented  on 
the  MEIMS  tool  is  relevant  to 
maintenance  operations 

□ 

□ 

□ 

□ 

□ 

(comments) 

The  concept  of  the  MEIMS 
tool  is  a  good  one. 

□ 

□ 

□ 

□ 

□ 

(comments) 
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Part  III.  User  Overall  Satisfaction  with  the  MEIMS  Prototype  Tool 

Please  make  any  comments  on  the  MEIMS  Prototype  Tool  not  reflected  in  your 
comments  in  sections  1  and  2. 

The  most  positive  aspects  of  the  MEIMS  prototype  tool  were: 


The  most  negative  aspects  of  the  MEIMS  prototype  tool  were: 


I  would  make  these  changes  (if  any)  to  the  MEIMS  prototype  tool: 


Thank  you!  Your  participation  is  greatly  appreciated! 
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